acceptodds
Under review as a conference paper at ICLR 2027

DeepSurvey: Agent-Oriented Automated Survey Generation with Analytical Depth and Citation Reliability

17 views

Abstract

As scientific literature grows rapidly and research increasingly involves AI agents, automated survey generation has become a key capability for both agents and human researchers. For such agents, a survey serves as a primary knowledge source of a field prior to research. Since pretrained models already encode broad knowledge of established work, these consumers benefit more from analytical depth than from breadth alone; moreover, unsupported claims, once ingested as knowledge, can propagate into downstream research. However, existing systems tend to overemphasize coverage and presentation; they suffer from limited analytical depth due to reliance on abstracts and isolated paper processing, and from unreliable citations due to imprecise retrieval and post-hoc grounding. We present DeepSurvey, an agentic generation system that addresses both limitations. To enhance depth, DeepSurvey extracts structured keynotes, models cross-paper relationships through clustering and comparative analysis, and integrates a code-agent subsystem to recover implementation-level details. To fortify reliability, it combines citation-graph expansion with hybrid filtering for topic-focused retrieval, enforces evidence-constrained analysis and writing, and deploys multi-granularity agentic refinement to validate citation–claim alignment. Experiments show that DeepSurvey achieves the highest content score (8.34/10) and citation quality (recall and precision gains of 25.3% and 35.2% over the strongest baseline), generalizes more robustly across domains (a CS-to-non-CS score drop of 0.16 vs. 0.28–0.46 for prior systems), and is preferred by domain experts over human-written surveys (83.3% in overall quality, 100% in content depth). Moreover, when a coding agent uses a survey as its only literature source, the agent equipped with DeepSurvey achieves the best performance among human-written and baseline-generated surveys.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Automated Survey Generation, LLM Agents, Citation Reliability, In-depth Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6AesOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents DeepSurvey, an agent-oriented framework for automated survey generation that emphasizes analytical depth and citation reliability. The framework integrates graph-based retrieval, full-text analysis, cross-paper relation modeling, and code-level analysis with evidence-constrained writing and multi-granularity refinement. Extensive experiments demonstrate improvements in content quality and citation reliability across multiple domains, as well as promising results on downstream research tasks. Although its methodological novelty is somewhat limited, the framework is comprehensive and supported by substantial empirical evaluation.

Critical strengths
  • Comprehensive literature analysis. DeepSurvey constructs a structured knowledge base at the paper, cluster, and survey levels. Full-text analysis, cross-paper relation graphs, comparison tables, and guided Q&A help capture technical connections across papers and improve survey depth.

  • Useful code-level analysis. The Code Agent Subsystem examines open-source repositories to recover implementation details that are insufficiently described in papers. This is particularly valuable for computer science surveys.

  • Citation reliability throughout the pipeline. Citation graph expansion, citation anchoring, evidence-constrained writing, and multi-agent revision collectively address citation reliability. Comparisons with SurveyForge support the effectiveness of this design.

  • Extensive evaluation. The experiments compare against four representative methods across five domains, with additional cross-domain evaluation and ablation studies. The paper also examines the reliability of its evaluation methodology.

  • Practical downstream value. The PINNacle experiments evaluate whether generated surveys can support autonomous research agents. The positive results provide encouraging evidence of their practical utility.

Critical weaknesses
  • Limited methodological novelty. The main contribution lies in integrating established techniques into a comprehensive system. While this integration is valuable, there is a lot of existing deepsurvey research. So the paper could more clearly distinguish its novel contributions from existing approaches.

  • Best-of-three reporting. Table 2 reports the best result across three runs rather than the average. Although Appendix C.5 explains this choice and applies it consistently, the results do not adequately characterize typical performance or variability. Reporting means and standard deviations would strengthen the evaluation.

  • Small-scale expert evaluation. The reported win rates against human-written surveys—83.3% for overall quality and 100% for content depth—are encouraging. However, the evaluation covers only six topics with four annotators each, yielding 24 paired judgments per dimension. This scale limits the generalizability of the conclusions, particularly claims of superiority over human-written surveys.

Minor issues:

  • In Section 2.3, “(details in Appendix A.1.” is missing a closing parenthesis.
  • Capitalization is inconsistent in Figure 3.
Questions for authors
  1. Could you further explain which components or interactions in DeepSurvey constitute the main methodological advances over existing survey generation frameworks?
  2. Could you report the mean and standard deviation across the three runs in Table 2?
  3. Given the limited scale of the expert evaluation, how robust are the reported win rates across topics and annotators? Could you provide more evidence supporting its robustness?
6 comments
AuthorsOpenReview ↗

Response to Reviewer 6Aes (Part 1 of 3)

Your careful and constructive comments have helped us improve the clarity and empirical presentation of our work, and we sincerely thank you for them. The revised manuscript has been uploaded, with the main revisions marked in blue.

1. Methodological advances beyond existing survey-generation frameworks (-> weakness 1, question 1)

We thank the reviewer for raising this important point. We first emphasize that our primary contribution and core motivation lie in being the first work to propose agent-oriented survey generation. Rather than focusing on the isolated construction of any single module, our paper centers on realizing agent-oriented survey generation around analytical depth and citation reliability—an objective thoroughly validated by both our rubric-based evaluations and downstream experiments. As automated research and recursive self-improvement (RSI) become increasingly important, we believe this problem formulation is both timely and novel, and that DeepSurvey provides useful infrastructure for subsequent research in these areas.

Beyond this novel problem formulation, DeepSurvey introduces methodological advances at both the component and system levels. We agree that the original manuscript did not explain these distinctions from prior methods in sufficient detail and will clarify them in the revision (Lines 935–963 in the revised manuscript).

System-level architectural designs and interactions.

Cross-domain framework design. To the best of our knowledge, DeepSurvey is the first survey-generation framework explicitly designed for cross-domain generalization, whereas most prior systems focus on a single domain. It integrates multi-domain reference sources and open-ended full-text extraction across retrieval, analysis, and generation, supporting cross-domain RSI and automated research.

Pipeline-wide citation-reliability control. Prior systems typically check citations through post-hoc refinement. In contrast, DeepSurvey coordinates source attribution and citation verification across analysis, outline construction, paragraph-level writing, and multi-granularity refinement, preventing citation errors from accumulating between stages.

Component-level innovations.

Citation-graph-based retrieval. Prior work mostly uses embedding-based querying or re-ranking, or employs graphs only for analysis. To the best of our knowledge, DeepSurvey is the first survey-generation framework to use citation-graph expansion as an integral literature-retrieval mechanism. As discussed in Lines 156-161, citation graphs inherently encode inter-paper relationships, making the retrieved literature better suited to downstream graph construction and clustering.

Code-analysis integration. To our knowledge, DeepSurvey is the first survey-generation system to introduce a dedicated code-agent subsystem. It examines official repositories and extracts implementation-level knowledge that is often omitted or condensed in papers, thereby providing concrete guidance for downstream research. The experiments (Lines 471–480) and case study (Lines 2741–2753) further demonstrate the effectiveness of this subsystem.

Tiered evidence-scoped writing. DeepSurvey organizes the evidence available to each writing unit into two tiers: a compact set of gold papers that provides the primary evidential basis and a broader set of relevant papers that supplies complementary context when needed. This design keeps generation anchored in a small number of high-value sources while retaining access to broader literature for synthesis and comparison, thereby reducing irrelevant context and hallucination risk[1][2].

The ablation study (Lines 426–470 and Appendix E) empirically confirms the contribution of each of these novel components to overall system performance.

[1] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024.

[2] Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, et al. Context length alone hurts LLM performance despite perfect retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 23281–23298, 2025.

AuthorsOpenReview ↗

Response to Reviewer 6Aes (Part 2 of 3)

2. Complete three-run results support DeepSurvey's overall advantage (-> weakness 2, question 2)

We thank the reviewer for this suggestion. We report the case-by-case results for all three runs, followed by their mean and standard deviation (Lines 1558–1609 in the revised manuscript).

2.1 Case by case and mean&std data

Table 1. Case-by-case PINNacle results for all three runs of each condition. Errors are relative L2L_2 (lower is better); X-factor, Novelty, and Insightfulness are higher-is-better.

ConditionRunBurgers-1D-CHeat-2D-CGHeat-2D-MSWave-1D-COverallX-factorNoveltyInsightfulness
DeepSurvey11.36e-22.64e-21.18e-29.54e-31.53e-211.18×5.676.00
DeepSurvey21.46e-22.74e-22.63e-29.54e-31.95e-28.81×5.005.00
DeepSurvey31.88e-22.60e-23.12e-55.10e-11.39e-11.24×5.005.00
Human-written survey12.20e-22.59e-24.80e-21.63e-16.48e-22.65×2.672.00
Human-written survey25.12e-21.63e-11.23e-11.68e-11.26e-11.36×4.334.00
Human-written survey31.56e-25.66e-27.70e-23.72e-11.30e-11.32×4.673.00
SurveyForge11.88e-22.60e-29.73e-25.10e-11.63e-11.05×1.001.00
SurveyForge21.88e-22.60e-29.73e-25.10e-11.63e-11.05×1.001.00
SurveyForge31.42e-26.77e-26.64e-25.41e-11.72e-11.00×4.334.67
No survey11.33e-21.78e-21.85e-24.68e-11.29e-11.33×4.002.67
No survey21.88e-22.60e-29.73e-25.10e-11.63e-11.05×1.001.00
No survey31.34e-21.66e-21.30e-26.22e-11.66e-11.03×4.673.67

The best run for each condition, selected by the lowest Overall error, is summarized below.

Table 2. Best-run PINNacle result per condition, where the best run is selected by the lowest Overall error.

ConditionBurgers-1D-CHeat-2D-CGHeat-2D-MSWave-1D-COverallX-factorNoveltyInsightfulness
DeepSurvey1.36e-22.64e-21.18e-29.54e-31.53e-211.18×5.676.00
Human-written survey2.20e-22.59e-24.80e-21.63e-16.48e-22.65×2.672.00
SurveyForge1.88e-22.60e-29.73e-25.10e-11.63e-11.05×1.001.00
No survey1.33e-21.78e-21.85e-24.68e-11.29e-11.33×4.002.67

Table 3. Three-run mean ± standard deviation of the PINNacle results per condition.

ConditionBurgers-1D-CHeat-2D-CGHeat-2D-MSWave-1D-COverallX-factorNoveltyInsightfulness
DeepSurvey1.567e-2 ± 2.253e-32.660e-2 ± 5.888e-41.271e-2 ± 1.074e-21.764e-1 ± 2.359e-15.793e-2 ± 5.735e-27.077×5.223 ± 0.3165.333 ± 0.471
Human-written survey2.960e-2 ± 1.550e-28.183e-2 ± 5.875e-28.267e-2 ± 3.088e-22.343e-1 ± 9.737e-21.069e-1 ± 2.984e-21.777×3.890 ± 0.8743.000 ± 0.816
SurveyForge1.727e-2 ± 2.168e-33.990e-2 ± 1.966e-28.700e-2 ± 1.457e-25.203e-1 ± 1.461e-21.660e-1 ± 4.243e-31.033×2.110 ± 1.5702.223 ± 1.730
No survey1.517e-2 ± 2.569e-32.013e-2 ± 4.177e-34.293e-2 ± 3.851e-25.333e-1 ± 6.500e-21.527e-1 ± 1.678e-21.137×3.223 ± 1.5962.447 ± 1.101
AuthorsOpenReview ↗

Response to Reviewer 6Aes (Part 3 of 3)

Values are reported as mean ± standard deviation over three runs. X-factor is the arithmetic mean of the three run-level factors,

2.2 Brief analyses on the case by case and mean&std data in PINN experiment

where each run-level factor is computed as the baseline error divided by that run's Overall error. We also found that the original Table 2 in the manuscript combined algorithmic novelty with diagnostic insightfulness; we therefore separated these dimensions, re-evaluated all runs, and updated Table 2 of the manuscript and Appendix C accordingly (Lines 357–389, 1358–1440, and 1558–1609).

The advantage is not limited to the selected best run. DeepSurvey achieves the lowest three-run mean Overall error (5.793e-2), 45.3% below the next-best condition, together with the highest mean X-factor, Novelty, and Insightfulness. The case-wise results also show a clear advantage on Heat-2D-MS and Wave-1D-C while remaining competitive on Burgers-1D-C and Heat-2D-CG, yielding the strongest aggregate performance across cases.

Note that some runs or individual cases have identical errors because unsuccessful modifications leave the vanilla implementation unchanged.

3. Internally consistent expert judgments and appropriately scoped conclusions (-> weakness 3, question 3)

We thank the reviewer for highlighting this limitation. We agree that six topics and 24 paired judgments per dimension are insufficient to establish broad superiority over human-written surveys across topics and domains. Our study was intended to provide initial high-quality evidence through blinded pairwise evaluation by researchers with relevant domain expertise, rather than large-scale crowdsourced evaluation. Expanding it reliably would require recruiting additional experts with research and publication experience in each new topic, which is difficult to complete within the rebuttal period. Nevertheless, our meta-evaluation shows substantial agreement among human experts: mean pairwise Cohen’s κ is 0.6958, and the all-human panel achieves Fleiss’ κ of 0.6578 (Section 3.6 and Appendix B.3). Although these agreement results do not increase the evaluation scale, they indicate that the expert judgments are internally consistent rather than driven by annotator noise. We will explicitly acknowledge the scale limitation and consider a broader expert evaluation in future work.

The current results therefore provide consistent but deliberately scoped evidence: DeepSurvey received 83.3% preference for overall quality and was preferred in all 24 judgments for Content Depth. Importantly, this human evaluation is supplementary; our primary evidence for DeepSurvey’s advantage in agent-oriented settings comes from the downstream agent experiments (Lines 348–410), which directly measure how effectively each survey supports research agents. To ensure academic rigor and appropriately scope our conclusion, we will revise the general claim “superior to human-written surveys” to the more precise statement “preferred by domain experts in our six-topic evaluation.” (Lines 421–422 in the revised manuscript)

4. Minor issues corrected

We thank the reviewer for the careful reading and for identifying these issues. We have corrected them in the revised manuscript.

Reviewer_9d3rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents DeepSurvey, an agent-oriented system for automated survey generation, focusing on analytical depth and citation reliability. The system combines graph-based literature retrieval, structured paper understanding, cross-paper analysis, and a Code Agent that extracts implementation details from code repositories. The authors conduct extensive experiments across multiple research domains, comparing DeepSurvey with existing survey generation methods. The results show improvements in content quality and citation reliability, supported by ablation studies, human evaluations, and downstream experiments where a coding research agent uses the generated surveys as its knowledge source.

Critical strengths
  1. The paper provides comprehensive evaluations across multiple domains, including comparisons with several existing open-source methods, ablation studies, human evaluations, and downstream experiments with a research agent. These experiments provide strong empirical support for the effectiveness of the proposed system.
  2. The performance is good. The system consistently outperforms existing baselines in content quality and citation reliability, with particularly notable improvements in citation precision and recall. The results are also supported by evaluations using human evaluation and different LLM judges.
Critical weaknesses
  1. According to the ablation results, the system relies heavily on existing components such as literature retrieval and writing, while the newly introduced Code Agent brings relatively small improvements to the overall performance. Although the Code Agent is useful for extracting implementation details, its contribution seems limited compared to the other components. Moreover, code repositories may not be available in many non-CS domains. It would be helpful to further analyze which components drive DeepSurvey's improvements in these domains, especially when code analysis is unavailable.

  2. Apart from the Code Agent, most of the components, including structured paper representations, cross-paper analysis, outline-guided writing, and multi-agent refinement, have already been explored in previous survey generation systems. DeepSurvey mainly combines these existing techniques into a more complete framework, but it is unclear what fundamentally new mechanisms or architectural ideas are introduced.The ablation studies mainly remove entire modules, rather than isolating the specific design choices that lead to performance improvements, which seem to be what the authors want to highlight.

Questions for authors

No more questions, all in weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 9d3r (Part 1 of 5)

We greatly appreciate your thoughtful and constructive comments, which have helped us clarify the novelty and empirical contributions of our work. The revised manuscript has been uploaded, with the main revisions marked in blue.

1. Methodological and architectural advances beyond existing components (-> weakness 2)

We apologize that the original manuscript did not articulate our methodological advances and their distinctions from prior work with sufficient clarity. We clarify them as follows (Lines 935–964 in the revised manuscript).

We thank the reviewer for raising this point. We agree that structured paper representations, cross-paper analysis, outline-guided writing, and multi-agent refinement have each been explored previously. However, this does not make DeepSurvey a simple integration of existing components: retrieval, analysis, and writing are universal stages, while our contribution lies in redefining their objectives and interactions for downstream research agents.

First, DeepSurvey introduces a new problem formulation and system objective. To the best of our knowledge, it is the first framework to explicitly formulate agent-oriented survey generation, prioritizing analytical depth, implementation specificity, and citation reliability over coverage and presentation alone. These requirements shape every stage of the pipeline and are validated directly through downstream agent experiments, rather than being implemented as a surface-level audience prompt.

Second, DeepSurvey introduces two system-level architectural advances. Unlike prior systems optimized primarily for a single domain, DeepSurvey treats cross-domain adaptation as a first-class requirement: multi-domain sources, open-ended full-text representations, and domain-agnostic analysis are integrated throughout the pipeline rather than added as task-specific templates. It likewise treats citation reliability as a pipeline-wide constraint rather than a post-hoc correction. Source attribution begins during analysis, citations are anchored during outline planning, every generated paragraph is verified against its assigned evidence, and multi-granularity refinement rechecks claim–citation alignment. This closed-loop design prevents citation errors from propagating across stages.

Third, DeepSurvey introduces several concrete component-level innovations. To the best of our knowledge, it is the first survey-generation framework to introduce (i) citation-graph-based retrieval as an integral retrieval mechanism, preserving inter-paper relationships for downstream clustering and analysis while hybrid filtering controls topical noise; (ii) a dedicated Code Agent subsystem, which recovers implementation-level knowledge absent or condensed in paper text; and (iii) tiered evidence-scoped writing, which assigns each subsection a compact set of gold papers plus broader relevant evidence, reducing context noise without losing useful comparisons [1,2]. DeepSurvey further strengthens these components through centralized multi-agent, multi-level refinement, where a planning agent maintains draft state, evidence, feedback, and revision history while coordinating review and revision across subsection, section, and survey levels. Its multi-view Analyzer combines global structural views from relation graphs and comparison tables with fine-grained reasoning from guided Q&A. As shown in Table 3 below, these analytical views yield substantially greater gains together than individually, providing direct evidence of coordinated interaction rather than component stacking.

The ablation studies further confirm that these innovations contribute positively to DeepSurvey’s effectiveness in agent-oriented settings (Table 3 below; Lines 426–470, 935–963, and 2396–2414).

[1] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173, 2024.

[2] Yufeng Du, Minyang Tian, Srikanth Ronanki, Subendhu Rongali, Sravan Babu Bodapati, et al. Context length alone hurts LLM performance despite perfect retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 23281–23298, 2025.

AuthorsOpenReview ↗

Response to Reviewer 9d3r (Part 2 of 5)

2. Contribution and scope of the Code Agent (-> weakness 1)

2.1 Direct empirical evidence of the Code Agent's contribution (-> weakness 1)

First, the direct comparison between DeepSurvey and DeepSurvey-noCode demonstrates the substantial contribution of the Code Agent subsystem. In the CS domain, the Code Agent improves Total Score by 0.18 and Content Depth by 0.41. It further raises Implementation Fidelity from 4.36 to 4.90 (+12.4%) and Practical Reproducibility from 3.65 to 4.33 (+18.5%). These consistent gains show that the Code Agent substantially deepens the survey and converts high-level methodological descriptions into faithful, reproducible implementation knowledge.

Second, ablation magnitudes across different modules do not directly measure their relative value. The appropriate comparison for the Code Agent is DeepSurvey with versus without code analysis. Components such as the collector and writer are indispensable backbone stages whose modification affects the entire downstream pipeline. Therefore, comparing raw ablation drops across interventions with fundamentally different scopes does not support the conclusion that the Code Agent contributes little.

Appendix G.3.2 illustrates this contribution concretely. The code-enhanced survey provides implementation details and pseudocode for partial-similarity scoring at each decoding step and hidden-state caching. It thereby turns conceptual methods into actionable implementation guidance for downstream research agents.

Third, to directly test whether code analysis benefits downstream agents, we evaluate DeepSurvey-noCode in the PINNacle downstream experiment (Lines 1611–1629 in the revised manuscript) across three runs:

Table 1. PINNacle best-run result per condition, where the best run is selected by the lowest Overall error. Errors are relative L2L_2 (lower is better); X-factor, Novelty, and Insightfulness are higher-is-better.

ConditionBurgers-1D-CHeat-2D-CGHeat-2D-MSWave-1D-COverallX-factorNoveltyInsightfulness
DeepSurvey1.36e-22.64e-21.18e-29.54e-31.53e-211.18×5.676.00
DeepSurvey-noCode1.38e-22.60e-29.40e-41.35e-14.39e-23.91×4.003.33
Human-written survey2.20e-22.59e-24.80e-21.63e-16.48e-22.65×2.672.00
SurveyForge1.88e-22.60e-29.73e-25.10e-11.63e-11.05×1.001.00
No survey1.33e-21.78e-21.85e-24.68e-11.29e-11.33×4.002.67

Table 2. Three-run mean ± standard deviation of the PINNacle results per condition.

ConditionBurgers-1D-CHeat-2D-CGHeat-2D-MSWave-1D-COverallX-factorNoveltyInsightfulness
DeepSurvey1.567e-2 ± 2.253e-32.660e-2 ± 5.888e-41.271e-2 ± 1.074e-21.764e-1 ± 2.359e-15.793e-2 ± 5.735e-27.077×5.223 ± 0.3165.333 ± 0.471
DeepSurvey-noCode2.003e-2 ± 4.633e-32.003e-2 ± 5.165e-32.368e-2 ± 1.877e-23.833e-1 ± 1.756e-11.120e-1 ± 4.814e-22.087×4.557 ± 0.4163.777 ± 0.316
Human-written survey2.960e-2 ± 1.550e-28.183e-2 ± 5.875e-28.267e-2 ± 3.088e-22.343e-1 ± 9.737e-21.069e-1 ± 2.984e-21.777×3.890 ± 0.8743.000 ± 0.816
SurveyForge1.727e-2 ± 2.168e-33.990e-2 ± 1.966e-28.700e-2 ± 1.457e-25.203e-1 ± 1.461e-21.660e-1 ± 4.243e-31.033×2.110 ± 1.5702.223 ± 1.730
No survey1.517e-2 ± 2.569e-32.013e-2 ± 4.177e-34.293e-2 ± 3.851e-25.333e-1 ± 6.500e-21.527e-1 ± 1.678e-21.137×3.223 ± 1.5962.447 ± 1.101

These results show that the Code Agent brings a decisive gain in downstream coding research:

  • Substantial error reduction. Adding the Code Agent cuts best-run Overall error by 65.1% (4.39e-2 → 1.53e-2, raising X-factor from 3.91× to 11.18×) and reduces three-run mean error by 48.3% (1.120e-1 → 5.793e-2, raising mean X-factor from 2.087× to 7.077×). On the hardest PDE (Wave-1D-C), best-run error drops by 14.2× (1.35e-1 → 9.54e-3).
  • Critical for surpassing published SOTA. In best-run performance, DeepSurvey-noCode (4.39e-2) already beats the human-written survey (6.48e-2) and SurveyForge (1.63e-1), yet still falls short of the published PINNacle SOTA (3.81e-2). Only with the Code Agent does the downstream agent surpass published SOTA (1.53e-2).
  • Stronger algorithmic novelty and deeper insights. Adding the Code Agent substantially elevates Novelty (4.00 → 5.67 best, +41.8%; 4.557 → 5.223 mean) and Insightfulness (3.33 → 6.00 best, +80.2%; 3.777 → 5.333 mean, +41.2%). Exposing concrete code-level mechanisms enables the downstream agent to pinpoint structural bottlenecks and invent novel methodological solutions rather than relying on superficial tuning.
AuthorsOpenReview ↗

Response to Reviewer 9d3r (Part 3 of 5)

2.2 Scope of Code-agent subsystem (-> weakness 1)

The Code Agent subsystem is specifically designed for code-rich domains such as computer science, so its concentrated gains in CS align directly with our design intent. Code repositories and comparable implementation-level artifacts (e.g., fine-grained experimental protocols or raw data) are less consistently available outside CS. Crucially, DeepSurvey retains a clear advantage even without code analysis. As shown in Table 4 of the revised manuscript (Section 3.5, Lines 446–470), the w/o code variant achieves 8.217 in Total Score and 7.655 in Content Depth, still clearly outperforming the strongest baseline, SurveyForge (8.071 / 7.485), and surpassing all other baselines by wide margins.

Reviewer_tVt1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DeepSurvey, an agent-oriented system for automated survey generation. The authors argue that research agents need surveys that provide not only broad literature coverage and clear explanations, but also in-depth mechanistic analysis, implementation details, and reliable claim-to-citation alignment. To address these needs, DeepSurvey uses a multi-stage workflow to process papers, analyze cross-paper relationships, generate a survey, and iteratively revise it. The authors evaluate the system through automatic evaluations, ablation studies, human reviews, and a downstream coding research task.

Critical strengths
  1. The research question has potential value. Exploring how research surveys can support agents in conducting follow-up research is worthwhile. Mechanistic analysis and implementation details in a survey may affect an agent’s choice of methods and experimental design.

  2. The system covers multiple stages of survey generation. DeepSurvey does more than generate text directly from retrieved results. It also includes literature analysis, cross-paper comparison, code information collection, citation checking, and iterative revision.

  3. The paper includes both human-survey comparisons and a downstream evaluation. The coding agent uses surveys from different sources as its literature input when optimizing PINNs on PINNacle. The paper also reports expert comparisons between DeepSurvey and human-written surveys. These evaluations move beyond text-only assessment, even though they do not isolate the effect of audience adaptation.

Critical weaknesses
  1. “Agent-oriented” remains a heuristic claim rather than an empirically established concept.
    The paper motivates agent-oriented survey generation with the hypothesis that agents need surveys emphasizing depth, specificity, and actionable future directions. It operationalizes this hypothesis through rubric changes and compares DeepSurvey with human-written surveys in expert evaluation and a downstream coding-agent task. However, the human-written surveys in these comparisons are not a controlled human-directed counterpart to DeepSurvey’s agent-oriented surveys. More broadly, these comparisons do not isolate the effect of audience adaptation from differences in survey quality, source, or generation system. Consequently, the observed gains may reflect the overall usefulness of DeepSurvey’s surveys rather than a benefit specific to tailoring surveys for agents.

    Appendix D.3 does specify agent-oriented scoring adjustments, including greater emphasis on research-relevant depth and specificity and a lower weight for writing quality. The concern is therefore not that the paper provides no operationalization. Rather, the relative distinctiveness of these criteria for agent consumers, and the choice of their weights compared with the needs of human readers, have not been validated in a controlled audience study. The paper offers a plausible design hypothesis, but does not yet establish empirically what distinguishes an agent-oriented survey from a human-oriented one, or whether that distinction is responsible for the reported performance gains. A controlled comparison between agent-directed and human-directed surveys, using the same generation system and evidence base as far as possible while varying the intended audience, would provide stronger evidence for the paper’s central claim. Evaluating each version with both human readers and downstream agents would help test whether audience and consumer-task matching matters.

  2. The downstream agent evaluation is promising, but its scope and reporting do not establish typical or stable gains.
    The paper evaluates one coding research task on PINNacle, covering four PDE cases. Each condition is run three times, but Table 2 reports only the best run. Appendix C.5 explains the authors’ rationale for selecting the best run. This choice can show the best performance reached under a condition, but it cannot establish the typical outcome or whether the differences between conditions are stable across runs. I encourage the authors to report all three runs, together with summary statistics such as the mean and variability, and to avoid drawing conclusions about stable gains from best-run results alone. The scope is also limited to one coding task and four cases, so additional downstream tasks would be needed to assess generalizability.

    This reporting concern is specific to the downstream agent experiment. The main survey-quality experiments report repeated-run means and standard deviations; they should not be characterized as lacking repeated-run statistics.

  3. The generation backbone is limited to one primary model, leaving generalizability unclear.
    The main generation system uses MiMo-V2.5, so it remains unclear whether DeepSurvey’s relative advantages extend to other generation-model families. The paper includes a local ablation with MiMo-V2-Flash and cross-family judge evaluations, but these do not constitute a full replication of the generation comparison with another backbone. This is a secondary limitation: it would be useful to test generalizability, but the more central issue is whether the reported gains arise specifically from audience adaptation.

Questions for authors

Please see the Critical Weaknesses section. In particular, can the authors compare agent-directed and human-directed surveys while holding the generation system and evidence base as constant as possible, and evaluate both versions with human readers and downstream agents? Could they also report all three downstream runs per condition, with summary statistics, so that readers can assess typical performance and run-to-run variability? Evaluation with additional generation backbones would further clarify generalizability, but the audience-adaptation comparison is the more direct test of the paper’s central claim.

10 comments
AuthorsOpenReview ↗

Response to Reviewer tVt1 (Part 1 of 7)

We thank you for your thorough and insightful comments, which have greatly helped us improve both the framing and empirical validation of our work. The revised manuscript has been uploaded, with the main revisions marked in blue.

1. Clarification of our central motivation (-> weakness 1 and question 1)

We apologize that the original manuscript did not state our motivation and intended scope with sufficient clarity. The questions raised appear to partly stem from a mismatch between our intended claim and an audience-adaptation interpretation. We clarify our motivation and scope below (Lines 935–964 in the revised manuscript).

1.1 Scope of our claim and the controlled comparison (-> weakness 1 and question 1)

We first clarify the scope of our claim. Our primary contribution and core motivation lie in proposing agent-oriented survey generation. Rather than studying audience adaptation in isolation or optimizing for human-reading preferences, our goal is to maximize the utility of generated surveys for downstream research agents. From this perspective, evidence sources, system architecture, and analytical depth are not confounding variables to be held fixed; they are precisely the design mechanisms through which DeepSurvey operationalizes agent-oriented generation.

Accordingly, our comparisons with prior systems and human-written surveys evaluate whether the complete DeepSurvey pipeline better supports downstream agents. Meanwhile, our human-expert evaluation serves purely as a supplementary quality check—confirming that DeepSurvey’s surveys are factually sound and valued by domain experts—rather than as a controlled study of human-reader preferences.

We nevertheless agree that comparing an agent-oriented mode against a human-oriented counterpart under a shared backbone provides useful evidence for our design motivation, while also increasing DeepSurvey’s practical value as an open-source system and broadening its range of applications. We reiterate that this experiment directly tests whether our agent-oriented design choices improve utility for downstream research agents; our paper does not attempt to establish a general theory of audience adaptation.

1.2 Human-oriented DeepSurvey experment (-> weakness 1 and question 1)

Agent-oriented and human-oriented surveys require different system behaviors, so reducing their difference to a surface-level audience prompt would be artificial. We instead hold the DeepSurvey infrastructure, literature pool, and generation backbone fixed while adjusting stage-level objectives toward human reading needs:

  • Retrieval and organization: broaden graph-based retrieval and use coarser thematic clusters to emphasize field coverage and background context.
  • Analysis and writing: emphasize conceptual organization, historical development, and broad coverage, with less emphasis on fine-grained mechanism comparison and reproduction-level knowledge.
  • Refinement: optimize for narrative flow, readability, and conciseness rather than maximizing the density of technical and implementation specifics.

We compare the two modes on the same topic subset using the same generation model.

Table 1. Rubric scores of DeepSurvey (agent-oriented and human-oriented modes) and two baselines on the shared topic subset. All methods are evaluated with DeepSeek-V4-Flash; higher is better for all metrics.

ModeTotalCore QualityWriting QualityContent DepthCitation RecallCitation Precision
DeepSurvey (agent-oriented)8.8989.5007.3309.0800.6880.581
DeepSurvey (human-oriented)7.7248.4207.4407.1700.6150.567
SurveyForge7.2367.9207.0006.6700.5300.465
AutoSurvey7.1888.0807.7806.0000.5280.453

Across both evaluations, agent-oriented DeepSurvey clearly outperforms its human-oriented counterpart: it achieves higher Total Score (8.898 vs. 7.724) and Content Depth (9.080 vs. 7.170), while reducing best-run and three-run mean PINNacle Overall error from 4.75e-2 to 1.53e-2 and from 1.058e-1 to 5.793e-2, respectively. These controlled results demonstrate that agent-oriented objectives improve both survey depth and downstream research utility (Tables 6 and 7 below; Lines 349–360 and 1558–1609 in the revised manuscript).

Note that MiMo-V2.5 has been taken offline from major API platforms. Given the time and cost constraints of the rebuttal period, we conducted this supplementary comparison with DeepSeek-V4-Flash. Using the same backbone, the comparison remains controlled.

AuthorsOpenReview ↗

Response to Reviewer tVt1 (Part 2 of 7)

1.3 Clarification on the need of "human evaluation on both version" (->weakness 1)

We agree that evaluating both versions with both consumer types would be useful for studying symmetric audience-task matching. However, that is not the claim of this paper. Our claim is directional and task-specific: agent-oriented design should improve utility for downstream research agents. The controlled downstream comparison directly tests this claim. A human-reader preference study would instead test whether human-oriented design is preferred by humans. This is a separate claim, one that we do not make and that is not required to establish our central result. We therefore focus on downstream agent performance, while leaving a full symmetric audience-task study to future work.

2. Empirical validation of our agent-oriented evaluation design (-> weakness 2 and question 1)

We thank the reviewer for raising this question. To reiterate, our goal is not a theoretical comparison between human-oriented and agent-oriented paradigms, but to build an effective agent-oriented survey generation system. Accordingly, what matters is whether our rubric accurately reflects a survey's utility for downstream agents. The most direct way to validate this is through downstream agent performance, rather than comparing against human-oriented evaluations.

2.1 Correspondence analyses of rubric score and downstream experiment

We agree that while our paper validates judge–human agreement, it lacks a direct consistency analysis between rubric scores and downstream task outcomes. Incorporating the controlled variants above, we analyze the alignment between rubric scores and PINNacle performance (both best-run and three-run mean; Lines 1655–1723 in the revised manuscript):

Table 2. Rubric scores of the four survey conditions used in the consistency analysis. Higher is better for all metrics.

ConditionTotalCore QualityWriting QualityContent DepthCitation PrecisionCitation Recall
Human-written survey7.5338.2507.6676.7500.2050.471
SurveyForge7.3009.0005.0006.7500.4310.478
DeepSurvey8.5339.0007.6678.5000.6470.703
DeepSurvey (human-oriented)7.8338.2508.6677.0000.6020.649

Note that this experiment is conducted using DeepSeek-V4-Flash. The reason is the same as above.

Using the rubric scores in Table 2 and the PINNacle results in Tables 6 and 7 (Section 3.1), we compute correlations across the four matched survey conditions. Higher X-factor is better, whereas lower Overall error is better; therefore, consistency appears as positive correlations for X-factor and negative correlations for Overall error.

Table 3. Spearman and Pearson correlations between each rubric dimension and PINNacle X-factor. Best uses each condition's lowest-Overall-error run, and Mean uses its three-run average; higher is better.

Rubric dimension vs. X-factorBest SpearmanBest PearsonMean SpearmanMean Pearson
Total Score+1.0000+0.9811+1.0000+0.9628
Content Depth+0.9487+0.9884+0.9487+0.9937
Writing Quality+0.8000+0.4012+0.8000+0.3296
Core Quality+0.0000+0.3835+0.0000+0.4465

Table 4. Spearman and Pearson correlations between each rubric dimension and PINNacle Overall error. Best uses each condition's lowest-Overall-error run, and Mean uses its three-run average; lower is better, so consistency corresponds to negative correlations.

Rubric dimension vs. Overall errorBest SpearmanBest PearsonMean SpearmanMean Pearson
Total Score-1.0000-0.8411-1.0000-0.9228
Content Depth-0.9487-0.6636-0.9487-0.8117
Writing Quality-0.8000-0.8867-0.8000-0.7412
Core Quality+0.0000+0.2998+0.0000+0.0708

To visualize this relationship, we plot the rubric Total Scores from Table 2 against the best-run and mean PINNacle results from Tables 6 and 7 in Figures 1 and 2. The plots show perfect rank consistency, near-linear alignment with X-factor, and strong negative alignment with Overall error.

Since openreview does not support uploading pictures in the rebuttal phase, we put Figure 1 and Figure 2 in the anonymous link: https://anonymous.4open.science/r/ICLR-2027-supplement-37D4/

Figure 1. Consistency between rubric Total Score and PINNacle X-factor across the four matched survey conditions. Left: best-run X-factor; right: three-run mean X-factor. Dashed lines show linear fits, and higher is better.

Figure 2. Consistency between rubric Total Score and PINNacle Overall error across the four matched survey conditions. Left: best-run Overall error; right: three-run mean Overall error. Dashed lines show linear fits, and lower is better.

AuthorsOpenReview ↗

Response to Reviewer tVt1 (Part 3 of 7)

2.2 Conclusions on our evaluation protocol and DeepSurvey's adavantages in agent-oriented settings

Three conclusions follow. First, Total Score and Content Depth align consistently with downstream performance under both outcome formulations and both reporting statistics. Total Score achieves perfect rank correlation throughout (Spearman ±1.0000), while Content Depth remains similarly strong (Spearman ±0.9487); their Pearson correlations reach 0.9628–0.9937 for X-factor and |0.6636–0.9228| for Overall error. This supports both the importance of analytical depth to downstream agents and the design of our aggregate score.

Second, Writing and Core Quality are less consistently associated with downstream utility. Writing has substantially weaker linear alignment with X-factor (Pearson 0.3296–0.4012), while Core Quality has zero rank correlation in all comparisons. This contrast supports assigning greater emphasis to Content Depth than to Writing Quality in an agent-oriented rubric; it also shows that the strong Total Score alignment is not driven by a generic quality dimension alone.

Finally, the rubric and downstream experiments provide complementary validation: the rubric predicts which surveys better support an agent, while PINNacle independently confirms that the emphasized criteria translate into stronger research performance. Together with our judge–human agreement analysis, these results support the effectiveness and rigor of our evaluation protocol.

Comments on the paper

3 comments
Area_Chair_zTruOpenReview ↗

Discussion Window Closing in 2 days. Please Respond to the Author's Rebuttal!

Dear Reviewers,

The authors have thoughtfully responded to your reviews and the discussion window is coming to a close on November 18th. Please take a minute to engage with their response and discuss whether it changes your opinions.

Thank you for your time and effort!

AuthorsOpenReview ↗

Summary of our work, reviewer feedback, and our responses (part 1/2)

Dear ACs, SACs, and PCs,

We sincerely thank you all for your time and insightful feedback during the review process. To present our situation as clearly and fairly as possible, we would like to briefly (1) outline our work and (2) summarize the reviews and explain how we have addressed the reviewers' concerns point by point in our author responses.

Outline of our work

Problem and Motivation (Lines 39-50, 136-138, 937-943) As AI agents increasingly participate in scientific research, automated surveys can serve as a valuable knowledge source to guide downstream agents. However, existing survey generation systems remain designed exclusively for human-reading preferences (emphasizing broad coverage and presentation) rather than downstream agent utility. Motivated by this limitation, we are the first to formulate agent-oriented survey generation and propose DeepSurvey to optimize the entire generation pipeline for autonomous research agents.

Challenges (Lines 51-80, 243-248) Realizing agent-oriented survey generation poses three key challenges: 1) Analytical depth: agents require fine-grained cross-paper comparisons and implementation-level mechanisms, yet existing systems rely on abstracts or rigid domain templates, process papers in isolation, and overlook code repositories. 2) Citation reliability: unsupported claims ingested by an agent can propagate into downstream research errors, whereas dense semantic retrieval and post-hoc citation attachment leave claims weakly grounded. 3) Agent-oriented evaluation: prior benchmarks focus narrowly on CS, saturate on coarse human-oriented criteria (Lines 1153-1169), and rarely test whether generated surveys improve downstream research agents.

Methods For the first challenge, we build a reusable, domain-agnostic analysis substrate via full-text keynotes, question-oriented clustering with multi-view relation analysis, and a Code Agent that recovers repository-level implementation details (Lines 162-205). For the second challenge, we enforce closed-loop citation control across the pipeline through citation-graph retrieval with hybrid filtering, tiered evidence-scoped writing with per-paragraph verification, and planner-driven multi-granularity refinement (Lines 154-161, 207-239). For the third challenge, we construct a five-domain evaluation protocol with an agent-oriented rubric and directly evaluate surveys as knowledge sources for autonomous coding agents (Lines 250-292, 348-406).

Conclusion 1) Across five domains, DeepSurvey consistently outperforms existing survey generation baselines in content quality, analytical depth, citation reliability, and cross-domain robustness (Lines 270-325), and is strongly preferred over human-written surveys by blinded domain experts (Lines 410-420). 2) When used as the sole literature source for an autonomous coding agent across two distinct research benchmarks (PINNacle and TransportROM), DeepSurvey consistently achieves the lowest modeling errors and highest algorithmic novelty and insightfulness, surpassing the best published result on PINNacle (Lines 348-410).

For each part above, we also provide the corresponding line numbers in our manuscript for your convenience. We kindly refer you to the manuscript for further details.

Current review status

Reviewers tVt1 and 6Aes currently rate the paper Weak Accept (3); we addressed their main concerns in the discussion. Reviewer 9d3r rates the paper 2 and has not replied after our rebuttal; our responses to their concerns on novelty, ablation granularity, Code Agent scope, and non-CS effectiveness without code are summarized in Part 2 Weaknesses and our responses below.

Manuscript revisions We uploaded the revision; additions are marked in blue:

  • Framing: Clarified the agent-oriented formulation and system/component advances (Lines 937-963)
  • Survey evaluation: Added human-oriented and cross-backbone comparisons; discussed existing expert agreement in the main text (Lines 349-360, 416-418, 965-989)
  • Downstream validation: Added TransportROM, validated our rubric through correlations with PINNacle, and added multi-run and extra-case results (Lines 373-410, 1558-1790)
  • Module analysis: Clarified and extended ablations, and added non-code case analysis (Lines 424-426, 1611-1629, 2396-2414, 3161-3185)

Summary of reviews and our responses

Strengths highlighted by the reviewers

  • Valuable research question and practical downstream utility in supporting autonomous research agents (tVt1, 9d3r, 6Aes)
  • Comprehensive multi-stage framework combining graph retrieval, full-text keynotes, cross-paper relation modeling, and repository-level code analysis (tVt1, 6Aes)
  • Systematic, pipeline-wide citation reliability yielding notable improvements in citation precision and recall (9d3r, 6Aes)
  • Extensive evaluation across multiple domains, baselines, LLM judges, blinded expert reviews, and downstream coding-agent tasks (tVt1, 9d3r, 6Aes)
AuthorsOpenReview ↗

Summary of our work, reviewer feedback, and our responses (part 2/2)

Weaknesses and our responses

Request for a controlled comparison between "agent-oriented" and "human-oriented" surveys (tVt1) Our core motivation is proposing agent-oriented survey generation for downstream research agents, rather than studying audience adaptation in isolation. Since agent-oriented and human-oriented generation require different architectures and stage designs, building human-oriented counterparts without changing the generation system is impractical, and evaluating DeepSurvey on agent-oriented tasks is sufficient for testing our claim. We nevertheless agree that this comparison supports our motivation and open-source utility, so we implemented and evaluated a human-oriented mode on our shared infrastructure and backbone (Lines 349-360, 965-989), where agent-oriented DeepSurvey consistently outperforms it.

Validation of the agent-oriented evaluation rubric (tVt1) To verify that our rubric and weights reflect actual agent utility rather than heuristics, we analyzed the correlation between rubric scores and PINNacle performance across matched conditions (Lines 373-399, 1655-1723). Total Score achieves perfect rank correlation with downstream performance and Depth aligns nearly linearly, while Writing and Core Quality correlate weakly, justifying our higher weight on Depth.

Lack of multi-run statistics in downstream evaluation (tVt1, 6Aes) We reported all three runs with means and STD on PINNacle, and separately evaluated Novelty and Insightfulness (Lines 1558-1609). DeepSurvey achieves the lowest mean error plus the highest mean X-factor, Novelty, and Insightfulness, confirming stable gains beyond the best run.

Downstream evaluation limited to one PINNacle task with four cases (tVt1) We further evaluated DeepSurvey on TransportROM, an offline reduced-order modeling benchmark without solver or network access (Lines 389-406, 1725-1790). DeepSurvey again achieves the lowest best-run and mean errors, lowest variance, and highest Novelty and Insightfulness across conditions. We also evaluated DeepSurvey on three more PINNacle PDE cases, observing consistent gains (Lines 1631-1653).

On methodological novelty beyond existing frameworks (9d3r, 6Aes) Our primary contribution is proposing agent-oriented survey generation, redefining pipeline objectives and stage interactions for downstream agents rather than human reading alone (Lines 937-943). At the system level, DeepSurvey introduces template-free cross-domain generalization and closed-loop citation control across the complete pipeline rather than post-hoc checking (Lines 945-950). Although isolated module design is not our main claim, we also introduce component-level innovations in each stage(Lines 952-963).

On the granularity of ablation studies (9d3r) We clarified that most w/o ablations isolate specific designs via baseline replacements rather than stage removals (Lines 426-470). We also further conducted fine-grained ablation experiments on the Analyzer (separating clustering, graphs/tables, and Q&A), confirming that global structure and fine-grained reasoning are complementary and perform best together (Lines 2396-2414).

Contribution of the Code-agent Subsystem (9d3r) The Code Agent targets code-rich fields like CS, so its concentrated CS gains in Depth, Implementation Fidelity, and Practical Reproducibility match our design intent (Lines 469-478). We also conducted a PINNacle ablation with vs. without the Code-agent Subsystem (Lines 1611-1629), which cuts best-run and mean errors by 65.1% and 48.3% and is decisive for beating published SOTA.

Effectiveness in non-CS domains without code (9d3r) Where repositories are sparse outside CS, w/o code still beats SurveyForge and DeepSurvey shows the smallest non-CS drop among all methods (Lines 320-325, 446-468). Case analyses of intermediate outputs further show how keynotes and multi-view analysis improve taxonomy construction and critical synthesis without code (Lines 3161-3185).

Generalizability across generation backbones (tVt1) Beyond evaluations across judge families (Lines 2221-2257), we evaluated DeepSurvey against SurveyForge and AutoSurvey using DeepSeek-V4-Flash as the backbone (Lines 976-989). DeepSurvey preserves the exact ranking and leads clearly in content quality, depth, and citation reliability.

Scale and robustness of the expert evaluation (6Aes) Meta-evaluation of our blinded expert study shows substantial inter-expert agreement (Cohen's κ = 0.6958, Fleiss' κ = 0.6578; Lines 416-418, 1283-1291), confirming that our high preference rates reflect reliable consensus rather than noise. Acknowledging the six-topic scale and that recruiting more experts is infeasible during the rebuttal period, we moderated our claim accordingly (Lines 417-418).

We thank all reviewers and committee members for their time and feedback, and hope this summary helps you review our work and responses.

Best regards,

Team 35698

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.