- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors collect 3,515 public documents covering 16 of CL-bench's 18 subcategories, with a preference for material from the previous six months. Sources include IETF RFCs, SEC filings, court opinions, arXiv papers, wikis, rulebooks and handbooks. GLM-5.2 rewrites the documents by renaming entities, changing numbers, renumbering sections and shuffling lists.
GLM-5.2 then generates up to three questions per document. Each question includes a persona or system prompt and at least seven rubrics, at least five of which refer to document values. A teacher, Qwen3.8-2.4T by default, answers from the rewritten document, and its native reasoning trace is retained for SFT.
A gap check keeps a sample only if its rubric pass rate is at least 0.6 with the document, at most 0.3 without it, and differs by at least 0.4. Rejected questions receive one regeneration attempt. The final dataset contains 9,625 samples.
SFT raises CL-bench from 13.7 to 22.8 and CL-bench Life from 8.4 to 12.6. Subsequent GSPO, using Qwen3.5-397B-A17B to judge rubric pass rate with the official CL-bench judge prompt, reaches 24.6 and 13.8. The paper compares teachers, studies the gap check and data scaling from 100 to 9,625 samples, tests RL reward variants, and breaks down performance by context provenance. A broader benchmark suite reports gains in long-context use, instruction following and reasoning, with code and knowledge performance roughly unchanged.
Critical strengths
(1) The data produce a large improvement on a difficult benchmark. With about 9.6k examples, CL-bench goes from 13.7 to 22.8 after SFT and 24.6 after RL, whereas RL alone reaches 16.9. This provides useful evidence for the SFT-then-RL ordering. More encouragingly, the largest gains are on fictional contexts, from 17.5 to 31.1 and then 32.9 in Table 5. That result supports transfer beyond memorizing the public source documents and is the strongest evidence here for improved use of context.
(2) Teacher benchmark performance does not predict student performance. Kimi-K3 scores 27.0 on CL-bench but produces a student scoring 19.7; Qwen3.8-2.4T scores 23.9 and produces the best student at 22.8. Table 8 shows that student reasoning length closely follows the teacher's, and Table 9 gives an example in which a short inherited trace misses a rubric. This is a useful finding for choosing distillation teachers, even though the underlying cause needs more investigation.
Critical weaknesses
(1) The contribution of document perturbation is not separated from the rest of the pipeline. The motivation for rewriting is reasonable, but longer teacher traces on rewritten documents do not establish that rewriting improves downstream learning. The compared questions are not identical, and trace length is only an indirect indicator of context use. A same-source, same-teacher SFT comparison without rewriting would be the most informative addition. The existing gap-check ablation does test filtering, so that component has some direct support; it does not answer the perturbation question. A small audit of rewritten-document consistency would also help, particularly for numeric totals and cross-references. The current results justify the complete recipe more strongly than they justify necessity of each step.
(2) The evaluation needs a clearer account of overlap, reproducibility and the cost of the RL increment. Training follows CL-bench's domain taxonomy and draws on similar public sources, making a source and near-duplicate overlap check appropriate. The gains on fictional contexts and other benchmarks are reassuring, but do not replace that check. The paper also needs an explicit plan for releasing data, generation prompts and training code. Finally, RL increases median thinking length from 7,626 to 10,481 tokens and answer length from 1,492 to 2,569. The added accuracy is useful, but comes with increased inference cost and some regressions on the broader suite. Those trade-offs should accompany the headline result.
Questions for authors
- Can you compare SFT with and without document rewriting, keeping source documents, teacher, question-generation procedure and gap check as similar as possible? A smaller controlled run would already be informative.
- What overlap exists between training documents and CL-bench or CL-bench Life, measured by source URL and near-duplicate content? How do results change if any overlapping sources are removed?
- Which model performs the gap check, what is the final yield after regeneration, and will the documents, questions, rubrics, prompts and code be released?
- Can you report evaluation sampling settings and uncertainty for the filtering and RL gains, together with response lengths or inference cost? Was the disclosed judge-prompt modification applied consistently to all model comparisons?
Official Comment by Authors
Thank you for the review.
In our preliminary experiments, documents crawled without rewriting, particularly older documents, often produced samples whose questions could be answered without the document. Consequently, only 13.1% of the samples passed the gap check. Since samples that do not pass the gap check will result in degraded performance (Section 4.1), the rewriting is a necessary step in the pipeline. Otherwise, we need to collect 7x more documents to get the same number of samples that pass the gap check. This is not feasible in practice.
No overlap source document exists between training documents and the test data. We check data contamination with n-gram matching and only find all matched results are texts like journal disclaimers and license text, which are not relevant to the task.
Following Qwen3.6-35B-A3B, we apply
temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0as the model inference setting. In section 4.2 we report the details of the RL experiments, including the response length. The judge-prompt is consistent. We do not run experiments multiple times due to the high cost (including training and evaluation), but we believe the uncertainty (variance) is negligible compare with the performance gain.