- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CSMuse, an idea-generation system for complex and interdisciplinary computer science research. The system has three main components: (1) a structured paper-summary dataset built from premier conferences across multiple ACM CCS domains, where each paper is normalized into five fields (Problem, Background, Design, Evaluation, Conclusion); (2) domain-specific knowledge graphs constructed from these summaries, used to retrieve traceable evidence for generated ideas; and (3) a cross-domain multi-agent workflow in which a seed idea is generated by a primary-domain agent, critiqued by a critic agent, revised independently by domain-specific agents using evidence from their respective knowledge graphs, and finally integrated and novelty-refined. The authors evaluate CSMuse on three domains (Computing Methodologies, Networks, Computer Systems Organization) using GPT-5.5 and DeepSeek-V4-Pro, comparing against AI Scientist and VIRSCI, with additional ablation studies on evidence grounding and cross-domain refinement.
Critical strengths
1、The paper accurately identifies and targets a real limitation in existing cross-domain idea-generation systems. The authors explicitly note that simply aggregating independently generated ideas does not ensure meaningful cross-domain transfer and may produce incoherent or technically incompatible combinations. CSMuse addresses this through domain-specific knowledge graphs, evidence retrieval, and cross-domain multi-agent refinement. This is consequential because it grounds the system's design in a clearly articulated failure mode of prior work, rather than proposing a generic multi-agent pipeline. 2、The structured paper-summary dataset and domain-specific knowledge graphs provide a traceable, reusable foundation for evidence-grounded idea generation. By normalizing papers into five structured fields and constructing graphs with explicit entity and relation types, the authors preserve mechanism-level design information that raw PDFs or abstracts often omit. This is consequential because it enables the evidence-grounding and retrieval components that distinguish CSMuse from purely parameter-based LLM idea generation.
Critical weaknesses
1、The benefits of cross-domain knowledge transfer are inconsistent across domains, and the paper does not adequately explain this. In the pairwise comparisons, the variant without cross-domain knowledge slightly outperforms the full system in Computing Methodologies (51% vs. 49%), and the "Refinement alone" variant also outperforms "Retrieval and Refinement" in CM and CSO. While the authors provide a finer-grained analysis using CD/HD and LJ scores, the overall evidence for cross-domain transfer in CM remains weak. This is consequential because cross-domain knowledge transfer is one of the paper's three core contributions, and a mixed result undermines the claim that CSMuse "consistently outperforms" baselines. 2、The paper does not report basic statistics about the constructed knowledge graphs, nor does it provide concrete examples of retrieved evidence being used in the final ideas. The paper claims to be "evidence-grounded," but without information on graph scale (number of nodes and edges), coverage, or extraction accuracy, and without examples showing how retrieved evidence actually shapes the revised ideas, the reader cannot verify that the retrieval component is doing meaningful work rather than simply providing additional context to the LLM. This is consequential because evidence grounding is the paper's second core contribution.
Questions for authors
1、The cross-domain ablation results in Computing Methodologies show that the variant without cross-domain knowledge slightly outperforms the full system. Can you explain why cross-domain transfer appears less effective in CM than in Net and CSO? Is this due to the nature of the domain, the quality of the retrieved evidence, or some other factor? 2、The Overall Idea Score combines the LLM-as-a-Judge score and the CD/HD ratio. Can you clarify the dimension and range of the CD/HD ratio, and explain why this particular combination was chosen over alternatives?
Response to Reviewer bctB
We thank the reviewer for the thoughtful assessment and constructive suggestions, and for recognizing the value of our structured knowledge representation and evidence-grounded cross-domain design. We have revised the manuscript, with changes highlighted in blue. Below, we address each concern and question.
Weakness 1: Inconsistent benefits across domains
To better characterize these differences, we have added paired analyses in Section 6.5 and a dimension-level comparison in Figure 6. These analyses clarify where cross-domain integration contributes most and how its novelty gains relate to technical soundness and feasibility. We discuss the results in our response to Question 1 below.
Weakness 2: Knowledge-graph statistics and concrete evidence-use examples
Thank you for highlighting the need to make the evidence-grounding process more transparent. We have expanded the description of the constructed knowledge graphs in the revised manuscript and added a worked example illustrating the complete workflow.
Q1: Why does cross-domain transfer appear less effective in Computing Methodologies than in Networks and Computer Systems Organization?
The CM result reflects a smaller benefit from cross-domain integration under the pairwise evaluation: the single-domain Retrieval and Refinement variant is preferred in 51% of comparisons, versus 49% for CSMuse. This near-even result contrasts with the stronger pairwise performance in Networks and CSO.
To understand this difference, we added a comparison across the six evaluation dimensions in Figure 6. CSMuse achieves higher average scores for novelty, mechanism specificity, and evaluation quality across all three domains, alongside lower scores for technical soundness and feasibility. The novelty improvements are +0.28, +0.32, and +0.35 points in CM, Networks, and CSO, respectively. Thus, cross-domain transfer improves the assessed novelty of CM ideas, although this improvement does not translate into a higher pairwise preference rate.
One possible explanation is that the usefulness of external knowledge depends on how naturally it connects to the target problem. Networks and CSO problems may offer more direct opportunities to combine architectural and system-level mechanisms from different domains. For some CM problems, incorporating external mechanisms may require additional assumptions or implementation complexity, offsetting their novelty benefits.
These results highlight the value of cross-domain transfer in generating more novel and detailed research ideas, while identifying technical soundness and feasibility as priorities for further development. In future work, we will implement and empirically evaluate these ideas to assess their technical soundness and practical potential.
Q2: What are the dimension and range of CD/HD, and why is it combined with the LLM-as-a-Judge score in this way?
Thank you for requesting this clarification. Both HD and CD are computed from cosine similarities:
Each is dimensionless and has a theoretical range of [-1,1].
We define the Overall Idea Score as OIS=5LJ×HDCD, where LJ/5 normalizes the five-point judge score. The two factors provide complementary perspectives: LJ assesses the proposed idea across six quality dimensions, while CD/HD considers its relationship to historical and contemporary literature. When CD and HD are positive, the ratio favors ideas that align with contemporary research while exhibiting less overlap with historical work.
We chose a multiplicative formulation so that the literature-based factor scales the normalized quality score, rather than contributing a separate additive term requiring a mixing weight. This is a heuristic composite measure, rather than a calibrated probability or an independently validated measure of scientific value. We also report LJ and the embedding-based analyses separately so that the results can be assessed beyond the aggregate OIS.