- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DATASTAMP, a domain-specific instruction data synthesis framework that uses logic graphs and transfer graphs to reuse reasoning patterns across domains. Experiments on medical and humanities benchmarks show competitive performance, with ablations supporting the benefits of structural retrieval and graph guidance.
Critical strengths
-
Generating diverse, high-quality instruction data from unlabeled domain documents addresses an important challenge in adapting LLMs to specialized domains.
-
The method achieves competitive performance across two domains. The expanded ablations in Appendix F.2 provide useful evidence that structural retrieval and explicit graph guidance contribute to downstream improvements.
Critical weaknesses
-
The authors use a randomly initialized, frozen R-GCN for structural retrieval, without showing that its embeddings reliably capture structural compatibility. Moreover, removing domain-specific vocabulary does not necessarily make the extracted reasoning patterns domain-independent. The downstream improvements are encouraging, but they do not directly validate the claimed structural transfer mechanism.
-
Previous AMR and semantic-graph-based question generation methods have already explored structural abstraction and graph-guided generation. DESIGNER also extracts and transfers question-design patterns across domains. The proposed dual-graph representation and retrieval pipeline appear to be an extension of these ideas, but the paper does not clearly demonstrate what fundamentally new capability they provide.
Questions for authors
-
Why should a randomly initialized, frozen R-GCN capture meaningful structural similarity? Have the authors compared it with standard graph similarity methods or evaluated its stability across different random seeds?
-
How do the authors verify that the extracted graphs are genuinely domain-independent, rather than simply removing domain-specific terminology? For example, can the source domain still be predicted from the transfer graphs?
-
The node correspondence is constructed through greedy type alignment, while the final projection relies on LLM prompting. How do the authors ensure that the retrieved reasoning structure is actually preserved? Would perturbing the graph structure or node mapping affect the results?
-
What is the key difference between DATASTAMP and existing AMR/semantic-graph-based question generation methods, particularly DESIGNER? Can the authors clarify which part of the proposed framework provides a new capability beyond prior work?
-
How does the method compare with direct document-based QA generation or exemplar-guided generation using the same teacher model, documents, and generation budget? This seems necessary to isolate the contribution of the proposed graph-based pipeline.
Dear Reviewer ksb7,
Thank you again for your thoughtful feedback. We have provided clarifications and additional experiments addressing your initial concerns. We would appreciate your thoughts on our responses.
Best regards, Authors