Highlight-Then-Summarize: Learning to Compress Evidence for Long-Context Understanding

Long-context understanding is a fundamental capability for large language models to reason over lengthy documents, multi-turn conversations, and code. However, large language models often struggle with irrelevant and redundant information in long contexts, where task-relevant evidence can be sparse and scattered across distant positions, limiting their ability to reliably solve long context tasks. To address this problem, we propose Highlight-Then-Summarize (H2S), a compress-then-reason paradigm that identifies question-relevant evidence from long contexts and summarizes it into a focused representation, enabling LLMs to reason with less interference from irrelevant and redundant information. We construct H2S-Dataset, comprising 6,647 examples from 11 benchmark families with an average context length of 43.9K tokens, and introduce H2S-Bench for evaluation. Trained on H2S-Dataset, H2S-14B achieves an average score of 32.60 on H2S-Bench, outperforming Qwen3.8-27B by 10.17 points and achieving leading performance among open-source models. Further analysis shows that H2S-14B achieves the highest Evidence-Summary Quality (ESQ) score and retains 97.1% of its 16K-budget performance with only a 4K output budget, indicating that H2S produces higher-quality yet more compact reasoning. Our contributions are threefold: (1) a compress-then-reason paradigm, H2S, that focuses reasoning on relevant evidence while suppressing irrelevant context; (2) H2S-Dataset, a diverse training set comprising 6,647 examples from 11 benchmark families to support learning under the H2S paradigm; and (3) a reinforcement learning method, H2S-RL, with process-level rewards for evidence selection and summary construction. We hope this work provides a practical approach to improving long-context reasoning through evidence-focused summary. Code and H2S-Dataset will be released at https://anonymous.4open.science/r/Highlight-Then-Summarize-3DF2.