acceptodds
Under review as a conference paper at ICLR 2027

Learning to Learn from Context: Synthetic Training from Perturbed Public Documents

Abstract

Real-world tasks often require large language models (LLMs) to learn from complex task-specific context rather than pretrained parametric knowledge. This capability remains a weakness of LLMs, while human annotation for such task contexts is expensive and difficult to scale. Public high-quality documents are an abundant alternative, but much of the public web has already been consumed during pretraining: training on such documents naively would reward memorization rather than context learning. In this work, we attempt to make use of high-quality public documents with small perturbations and empirically find that LLMs can successfully generate context-dependent reasoning traces and answers, which are then used to train a student model. Specifically, we construct a synthesis pipeline that (i) rewrites source documents to reduce memorization risk, (ii) generates questions and rubrics that require reasoning over the document, (iii) answers the questions with the document as context, and (iv) admits only samples that genuinely depend on the document. Without any human annotators, our pipeline generates about 10k samples from 3.5k documents, and the resulting student model substantially improves the performance on CL-bench. SFT raises a Qwen3.6-35B-A3B student from 13.7% to 22.8%, and a subsequent rubric-reward RL stage reaches 24.6%, on CL-bench comparable with a frontier model of over a trillion parameters, Qwen3.8-2.4T (23.9%). We also observe a broad transfer of improvements to long-context understanding, instruction following, and reasoning, while code generation and knowledge remain mostly flat. We hope this work provides a reproducible and scalable way to improve the ability of LLMs to learn from context, and to facilitate further research on context-grounded reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
context learning, data pipeline, language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zo7MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors collect 3,515 public documents covering 16 of CL-bench's 18 subcategories, with a preference for material from the previous six months. Sources include IETF RFCs, SEC filings, court opinions, arXiv papers, wikis, rulebooks and handbooks. GLM-5.2 rewrites the documents by renaming entities, changing numbers, renumbering sections and shuffling lists.

GLM-5.2 then generates up to three questions per document. Each question includes a persona or system prompt and at least seven rubrics, at least five of which refer to document values. A teacher, Qwen3.8-2.4T by default, answers from the rewritten document, and its native reasoning trace is retained for SFT.

A gap check keeps a sample only if its rubric pass rate is at least 0.6 with the document, at most 0.3 without it, and differs by at least 0.4. Rejected questions receive one regeneration attempt. The final dataset contains 9,625 samples.

SFT raises CL-bench from 13.7 to 22.8 and CL-bench Life from 8.4 to 12.6. Subsequent GSPO, using Qwen3.5-397B-A17B to judge rubric pass rate with the official CL-bench judge prompt, reaches 24.6 and 13.8. The paper compares teachers, studies the gap check and data scaling from 100 to 9,625 samples, tests RL reward variants, and breaks down performance by context provenance. A broader benchmark suite reports gains in long-context use, instruction following and reasoning, with code and knowledge performance roughly unchanged.

Critical strengths

(1) The data produce a large improvement on a difficult benchmark. With about 9.6k examples, CL-bench goes from 13.7 to 22.8 after SFT and 24.6 after RL, whereas RL alone reaches 16.9. This provides useful evidence for the SFT-then-RL ordering. More encouragingly, the largest gains are on fictional contexts, from 17.5 to 31.1 and then 32.9 in Table 5. That result supports transfer beyond memorizing the public source documents and is the strongest evidence here for improved use of context.

(2) Teacher benchmark performance does not predict student performance. Kimi-K3 scores 27.0 on CL-bench but produces a student scoring 19.7; Qwen3.8-2.4T scores 23.9 and produces the best student at 22.8. Table 8 shows that student reasoning length closely follows the teacher's, and Table 9 gives an example in which a short inherited trace misses a rubric. This is a useful finding for choosing distillation teachers, even though the underlying cause needs more investigation.

Critical weaknesses

(1) The contribution of document perturbation is not separated from the rest of the pipeline. The motivation for rewriting is reasonable, but longer teacher traces on rewritten documents do not establish that rewriting improves downstream learning. The compared questions are not identical, and trace length is only an indirect indicator of context use. A same-source, same-teacher SFT comparison without rewriting would be the most informative addition. The existing gap-check ablation does test filtering, so that component has some direct support; it does not answer the perturbation question. A small audit of rewritten-document consistency would also help, particularly for numeric totals and cross-references. The current results justify the complete recipe more strongly than they justify necessity of each step.

(2) The evaluation needs a clearer account of overlap, reproducibility and the cost of the RL increment. Training follows CL-bench's domain taxonomy and draws on similar public sources, making a source and near-duplicate overlap check appropriate. The gains on fictional contexts and other benchmarks are reassuring, but do not replace that check. The paper also needs an explicit plan for releasing data, generation prompts and training code. Finally, RL increases median thinking length from 7,626 to 10,481 tokens and answer length from 1,492 to 2,569. The added accuracy is useful, but comes with increased inference cost and some regressions on the broader suite. Those trade-offs should accompany the headline result.

Questions for authors
  1. Can you compare SFT with and without document rewriting, keeping source documents, teacher, question-generation procedure and gap check as similar as possible? A smaller controlled run would already be informative.
  2. What overlap exists between training documents and CL-bench or CL-bench Life, measured by source URL and near-duplicate content? How do results change if any overlapping sources are removed?
  3. Which model performs the gap check, what is the final yield after regeneration, and will the documents, questions, rubrics, prompts and code be released?
  4. Can you report evaluation sampling settings and uncertainty for the filtering and RL gains, together with response lengths or inference cost? Was the disclosed judge-prompt modification applied consistently to all model comparisons?
1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the review.

Can you compare SFT with and without document rewriting, keeping source documents, teacher, question-generation procedure and gap check as similar as possible? A smaller controlled run would already be informative.

In our preliminary experiments, documents crawled without rewriting, particularly older documents, often produced samples whose questions could be answered without the document. Consequently, only 13.1% of the samples passed the gap check. Since samples that do not pass the gap check will result in degraded performance (Section 4.1), the rewriting is a necessary step in the pipeline. Otherwise, we need to collect 7x more documents to get the same number of samples that pass the gap check. This is not feasible in practice.

What overlap exists between training documents and CL-bench or CL-bench Life, measured by source URL and near-duplicate content? How do results change if any overlapping sources are removed?

No overlap source document exists between training documents and the test data. We check data contamination with n-gram matching and only find all matched results are texts like journal disclaimers and license text, which are not relevant to the task.

Which model performs the gap check, what is the final yield after regeneration, and will the documents, questions, rubrics, prompts and code be released?

  • See L240, we use GLM-5.2 for question generation and this includes the gap check.
  • The final yield after regeneration is 2660 samples.
  • Unfortunately, we are not able to provide the full dataset or code at this time.

Can you report evaluation sampling settings and uncertainty for the filtering and RL gains, together with response lengths or inference cost? Was the disclosed judge-prompt modification applied consistently to all model comparisons?

Following Qwen3.6-35B-A3B, we apply temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0 as the model inference setting. In section 4.2 we report the details of the RL experiments, including the response length. The judge-prompt is consistent. We do not run experiments multiple times due to the high cost (including training and evaluation), but we believe the uncertainty (variance) is negligible compare with the performance gain.

Reviewer_KYubOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds context-learning data by rewriting public documents, generating questions and rubrics, and keeping examples where the teacher performs much better with the document than without it. The results are substantial: 9,625 examples raise a Qwen3.6-35B-A3B student's CL-bench accuracy from 13.7% to 22.8%, and rubric-reward RL brings it to 24.6%.

Critical strengths
  1. The problem is worth addressing, and the recipe is clear. A teacher that already knows a public document may not need to use the supplied context. Perturbing realistic documents and checking answers with and without context is a practical response. Keeping the rubrics also makes the data reusable for both SFT and RL.

  2. The paper includes helpful controls and reports negative results. The gap-check comparison uses two datasets of 3,000 examples each (Table 3), and the scaling experiment examines the effect of data size. The discussion of teacher dependence, longer RL outputs, more anti-hallucination rubric failures, and ARC regressions is also useful. It gives a more realistic view of what this training achieves.

Critical weaknesses

From a high-level perspective, my concern is that the paper currently reads more like a benchmark-oriented data recipe than a study that offers a clear new insight into context learning. The gains are useful, but it is still unclear which part of the recipe matters, why it works, and how far the benefit extends beyond the task distribution used to design the data.

Questions for authors
  1. How far does the benefit generalize? The results on CL-bench Life and other benchmarks are encouraging. Since the training sources follow CL-bench's taxonomy, do the gains also hold on task categories deliberately excluded from data construction? This would help establish what transfers beyond the distribution used to design the recipe. For example, I'm curious whether the gains carry over to tasks where context accumulates and changes over time, such as multi-turn agents, conversational consistency, or repository-level bug fixing, given the model is now better at context understanding.

  2. What does rewriting add? The gap-check ablation is helpful, but I'm curious how much document rewriting itself contributes. Could you compare original and rewritten documents under matched teacher, filtering, and training-token budgets, alongside ordinary long-document QA distillation?

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the review.

How far does the benefit generalize? The results on CL-bench Life and other benchmarks are encouraging. Since the training sources follow CL-bench's taxonomy, do the gains also hold on task categories deliberately excluded from data construction? This would help establish what transfers beyond the distribution used to design the recipe. For example, I'm curious whether the gains carry over to tasks where context accumulates and changes over time, such as multi-turn agents, conversational consistency, or repository-level bug fixing, given the model is now better at context understanding.

It turns out that CL-bench itself contains exactly the held-out probe you ask for. The two categories deliberately excluded from data construction are exactly multi-turn agent tasks: Workflow Orchestration (229 tasks) and Operational Procedures (185 tasks). We explicitly report the results on these two categories below.

Task groupnBaselineSFTSFT+RLΔ SFTΔ SFT+RL
All tasks1,89913.722.824.6+9.1+10.9
16 subcategories in data construction1,48514.022.023.9+7.9+9.9
Workflow Orchestration22915.328.832.3+13.5+17.0
Operational Procedures1859.722.221.1+12.4+11.3
WO + OP combined41412.825.927.3+13.1+14.5

Clearly, the gains carry over to multi-turn agent tasks, and the gains are even larger than the average gain across subcategories that we constructed the data from.

What does rewriting add? The gap-check ablation is helpful, but I'm curious how much document rewriting itself contributes. Could you compare original and rewritten documents under matched teacher, filtering, and training-token budgets, alongside ordinary long-document QA distillation?

In our preliminary experiments, documents crawled without rewriting, particularly older documents, often produced samples whose questions could be answered without the document. Consequently, only 13.1% of the samples passed the gap check. Since samples that do not pass the gap check will result in degraded performance (Section 4.1), the rewriting is a necessary step in the pipeline. Otherwise, we need to collect 7x more documents to get the same number of samples that pass the gap check. This is not feasible in practice.

From a high-level perspective, my concern is that the paper currently reads more like a benchmark-oriented data recipe than a study that offers a clear new insight into context learning. The gains are useful, but it is still unclear which part of the recipe matters, why it works, and how far the benefit extends beyond the task distribution used to design the data.

That is a fair point. The reason why it reads like a benchmark-oriented work is that we mainly focus on context learning and we do not find other benchmarks that are as suitable as CL-bench for evaluating context learning. Context learning requires the model to "Learning from complex context and reasoning using new knowledge". Existing benchmarks 1) do not exclude the possibility of memorization; 2) do not contain intricate constraints that models must acquire from the provided information. e.g. LongBench v2 only contains multiple-choice questions, AA-LCR only need short answers. These are far from the real-world scenarios.

Nevertheless, we evaluate our method on common benchmarks and report the results in Table 2 and Table 7. The gains are consistent across all benchmarks. In section 4 we provide necessary ablations to analysis the effectiveness of our approach.

Reviewer_z7nPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to improve the ability of LLMs to learn and reason from task-specific contexts rather than relying on memorized knowledge. The authors propose a synthetic data generation pipeline based on perturbed public documents. The pipeline rewrites documents to reduce memorization, generates document-grounded questions and evaluation rubrics, and applies a gap check to retain samples that genuinely require contextual information.

Critical strengths
  • The paper addresses an important and practical problem of improving LLMs' ability to learn from complex contexts. The proposed data synthesis pipeline is straightforward and combines document rewriting with context-dependence filtering to construct targeted training data without human annotation.

  • The experimental results demonstrate substantial improvements on CL-bench and positive transfer to several other benchmarks. The gap-check ablation, data scaling experiments, and analyses across different teacher models provide additional evidence for the effectiveness of the proposed training pipeline.

Critical weaknesses
  • Deep rewriting is a central component of the proposed pipeline, intended to reduce memorization and encourage context-dependent reasoning. However, the paper does not directly evaluate its contribution to downstream student performance. Appendix C.2 only analyzes changes in teacher reasoning traces, with different questions generated for the original and rewritten documents. A controlled ablation comparing training on original versus rewritten documents under the same data and training budgets is needed to establish the effectiveness of this component.

  • The main experiments primarily compare the proposed pipeline with the original student model and different SFT/RL configurations, without direct comparisons against existing long-context synthetic data generation methods. Related approaches such as LongFaith [1], MIMG [2], and CLIPPER [3] also aim to improve long-context understanding and reasoning through synthetic supervision. Comparisons with these methods under matched student models, data sizes, and training budgets would help establish the advantages of the proposed pipeline.

[1] LongFaith: Enhancing Long-Context Reasoning in LLMs with Faithful Synthetic Data. Findings of ACL 2025.

[2] What Are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices. ACL 2025.

[3] CLIPPER: Compression Enables Long-Context Synthetic Data Generation. COLM 2025.

Questions for authors

Please refer to the concerns raised in the Critical Weaknesses section.

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the review.

Deep rewriting is a central component of the proposed pipeline, intended to reduce memorization and encourage context-dependent reasoning. However, the paper does not directly evaluate its contribution to downstream student performance. Appendix C.2 only analyzes changes in teacher reasoning traces, with different questions generated for the original and rewritten documents. A controlled ablation comparing training on original versus rewritten documents under the same data and training budgets is needed to establish the effectiveness of this component.

In our preliminary experiments, documents crawled without rewriting, particularly older documents, often produced samples whose questions could be answered without the document. Consequently, only 13.1% of the samples passed the gap check. Since samples that do not pass the gap check will result in degraded performance (Section 4.1), the rewriting is a necessary step in the pipeline. Otherwise, we need to collect 7x more documents to get the same number of samples that pass the gap check. This is not feasible in practice.

The main experiments primarily compare the proposed pipeline with the original student model and different SFT/RL configurations, without direct comparisons against existing long-context synthetic data generation methods. Related approaches such as LongFaith [1], MIMG [2], and CLIPPER [3] also aim to improve long-context understanding and reasoning through synthetic supervision. Comparisons with these methods under matched student models, data sizes, and training budgets would help establish the advantages of the proposed pipeline.

Thank you for the suggestion. We respectfully note that LongFaith, MIMG, and CLIPPER differ from our work primarily in the capability they target. LongFaith and MIMG synthesize supervision for multi-hop factoid QA, while CLIPPER targets narrative claim verification over books. Context learning requires the model to exclude memorization and learn to follow intricate constraints from the provided information [1]. These are beyond the scope of the above works.

Though we are willing to run the experiments, it seems that this is not feasible during the rebuttal period. The judge prompt for CL-bench is no longer working due to OpenAI's restriction, thus we can no longer run the evaluation. Therefore, we cannot provide a direct comparison with these methods at this time unfortunately.

[1] CL-bench: A Benchmark for Context Learning, https://arxiv.org/abs/2602.03587.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.