Context-CoT

While LLMs excel at reasoning over prompts 002 using static pre-trained knowledge, they strug- 003 gle significantly with context learning—the 004 ability to dynamically extract, internalize, and 005 apply new knowledge from complex, task- 006 specific contexts. Recent evaluations on the CL- 007 Bench reveal a critical capability gap: frontier 008 models solve only 17.2% of context-dependent 009 tasks on average. To bridge this gap, we pro- 010 pose Context-CoT, a novel Chain-of-Thought 011 (CoT) data synthesis and fine-tuning frame- 012 work specifically designed to enhance context 013 learning in open-source LLMs. We construct a 014 high-quality, context-grounded CoT dataset us- 015 ing a newly introduced three-stage pipeline: (i) 016 multi-stage CoT sampling, which first guides 017 the model to distill the long context into task- 018 relevant intermediate representations before 019 reasoning over the extracted contextual evi- 020 dence; (ii) rubric-based minimum-leakage fil- 021 tering, which hides reference answers and full 022 rubrics during CoT generation, provides only 023 minimal failed-rubric feedback when necessary, 024 and filters out trajectories that violate context- 025 specific criteria; and (iii) student-aware CoT 026 selection, which ensures the retained CoT paths 027 align naturally with the target model’s distribu- 028 tion for optimal learning efficiency. Extensive 029 experiments demonstrate that open-source mod- 030 els fine-tuned on our dataset achieve significant 031 performance gains on CL-Bench, substantially 032 reducing context-neglect errors. Our work pro- 033 vides a scalable, data-driven solution to transi- 034 tion open-source models from simple prompt 035 followers to robust context learners.