- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper presents RiNoGen, a pipeline to build long-form SFT data from a domain corpus. Seed entities are selected with clustering plus an LLM filter, expanded into "Rich-Node" graphs grounded on retrieved chunks, and the answers are synthesized with a Map-Reduce step, then the question is regenerated from the answer. Exps. on a large Chinese educational corpus with Qwen3-8B report a gain of about 1 pp over several corpus-to-data baselines on a 427-item MCQ subset.
Critical strengths
The problem (turning domain corpora into grounded SFT data) is relevant and the pipeline is described clearly. The internal controls (parametric, single chunk, flat RAG, graph) are reasonable.
Critical weaknesses
-
Weak positioning w.r.t. SOTA. The main building blocks are presented as new but they are quite established. Generating long SFT answers by decomposing them into sections is close to LongWriter/AgentWrite, retrieval + outline for long grounded text is close to STORM, and regenerating the question from the answer is basically instruction backtranslation (Humpback). Map-Reduce over summaries is the core of GraphRAG global search, which the paper cites only for the community summaries. None of these is discussed. Also the paper assumes that SFT transfers corpus knowledge to the model, but there is a known line of work showing that SFT mostly teaches the model to use what it already knows and that new facts in SFT data can increase hallucinations (e.g. LIMA, Gekhman et al. 2024). Using the same model as generator and student is related to self-distillation fine-tuning, and the data selection/curation literature is not considered either. As it is, it is hard to see what is really new apart from the combination of known pieces.
-
Relevant works omitted from the comparisons. The baselines are mostly generic text transformation methods, and some of them (EntiGraph, WRAP) are designed for (continued) pretraining, so a 3k-example SFT budget is not their natural setting. The most direct competitors for adapting a model from unlabeled domain corpora are missing, e.g. AdaptLLM (reading comprehension), Instruction Pre-Training and Bonito. Simple but strong baselines are missing too: a plan-then-write long-form generator (see AgentWrite), and the fine-tuned model used with RAG on the same index. For the web part, the obvious comparison RiNoGen-Web vs naive web RAG is not reported.
-
Experimental analysis is not fully convincing. The main gain (1.14 pp on 427 items, i.e. about 5 questions) is within the statistical noise. No significance test is given, and repeating the eval 20 times at temperature 0 does not measure any variance. The same method gets about the same difference as the claimed gain in two different settings. The external comparison is confounded, since RiNoGen uses the full corpus index while the baselines get a 7k chunk slice, and RiNoGen answers are much longer so token budgets are not matched (length alone is known to affect SFT results). The benchmark is generated by a same-family LLM from the same corpus without human validation. Long-form quality and provenance, which are the main selling points, are never evaluated directly. The web eval relies on an unspecified LLM judge and the result (web access helps on 2025-26 topics) is expected.
Questions for authors
- How does RiNoGen compare with AdaptLLM (https://arxiv.org/abs/2309.09530), Instruction Pre-Training(https://arxiv.org/abs/2406.14491), Bonito (https://arxiv.org/abs/2402.18334)?
- Can you report results with the same corpus access and matched token budgets for all methods?
- Can you add paired significance tests?