acceptodds
Under review as a conference paper at ICLR 2027

Highlight-Then-Summarize: Learning to Compress Evidence for Long-Context Understanding

Abstract

Long-context understanding is a fundamental capability for large language models to reason over lengthy documents, multi-turn conversations, and code. However, large language models often struggle with irrelevant and redundant information in long contexts, where task-relevant evidence can be sparse and scattered across distant positions, limiting their ability to reliably solve long context tasks. To address this problem, we propose Highlight-Then-Summarize (H2S), a compress-then-reason paradigm that identifies question-relevant evidence from long contexts and summarizes it into a focused representation, enabling LLMs to reason with less interference from irrelevant and redundant information. We construct H2S-Dataset, comprising 6,647 examples from 11 benchmark families with an average context length of 43.9K tokens, and introduce H2S-Bench for evaluation. Trained on H2S-Dataset, H2S-14B achieves an average score of 32.60 on H2S-Bench, outperforming Qwen3.8-27B by 10.17 points and achieving leading performance among open-source models. Further analysis shows that H2S-14B achieves the highest Evidence-Summary Quality (ESQ) score and retains 97.1% of its 16K-budget performance with only a 4K output budget, indicating that H2S produces higher-quality yet more compact reasoning. Our contributions are threefold: (1) a compress-then-reason paradigm, H2S, that focuses reasoning on relevant evidence while suppressing irrelevant context; (2) H2S-Dataset, a diverse training set comprising 6,647 examples from 11 benchmark families to support learning under the H2S paradigm; and (3) a reinforcement learning method, H2S-RL, with process-level rewards for evidence selection and summary construction. We hope this work provides a practical approach to improving long-context reasoning through evidence-focused summary. Code and H2S-Dataset will be released at https://anonymous.4open.science/r/Highlight-Then-Summarize-3DF2.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context understanding, question-conditioned summarization, large language model, context compression, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_piKaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Highlight-Then-Summarize (H2S), a compress-then-reason paradigm for long-context understanding: a single autoregressive model first emits source-grounded evidence (block IDs plus supporting spans) from a block-structured document, then a question-conditioned summary that integrates the selected evidence, and finally the answer. To supervise both intermediate stages, the authors construct H2S-Dataset (6,647 examples from 11 benchmark families, retro-annotated with evidence and summaries via semantic blocking, hybrid retrieval, question decomposition, claim extraction, and faithfulness/coverage/answerability quality checks) and H2S-RL, a GRPO objective with programmatic process rewards combining answer correctness, evidence grounding and span-level F1, summary ROUGE-L, and a structure gate, aggregated via a harmonic mean. Trained from Qwen2.5-7B/14B-Instruct-1M with a 32K→64K→128K curriculum, H2S-14B attains the best average (32.60) among the evaluated open-source models on H2S-Bench (seven tasks, 128K input / 4K output). Process-level analyses (the ESQ metric, component ablations, output-budget studies) support the intended mechanism: explicit compression substitutes for long generation, retaining 97.1% of the 16K-budget performance under a 4K budget.

Critical strengths

S1 — Supervising a genuinely under-supervised stage: evidence integration. As the paper's related-work discussion acknowledges, existing long-context RL methods reward evidence selection or final answers (e.g., LongRLVR's context reward targets selection; LoongRL's reasoning pattern emerges from outcome rewards), leaving the integration of dispersed evidence implicit in answer generation. H2S externalizes this stage as an observable intermediate representation and supervises it with process-level rewards whose harmonic-mean design prevents either component from masking the other. The ablations support the design: removing the summary costs 8.5–9.9 points and removing evidence 4.0–4.3 (Table 3).

S2 — Mechanism-transparent evaluation and honest reporting. The paper evaluates intermediate quality (ESQ, Table 5), isolates where gains originate (output-budget analysis: thinking baselines improve by 11–25 points when allowed 16K output while H2S improves by ≤1, Table 4), and analyzes failure cases at the stage level, distinguishing evidence-selection errors from arithmetic errors (Appendix A.6/A.9). The reported numbers are internally consistent across the abstract, main tables, and appendices, and limitations are stated explicitly.

Critical weaknesses

W1 — Train/eval benchmark-family overlap without an ID/OOD breakdown. Four of the seven H2S-Bench tasks (MRCR, Frames, DocFinQA, LongCite) also appear among the 11 H2S-Dataset training families (Figure A6), and Table 2 does not separate in-distribution from OOD performance. As reported, the headline average cannot be unambiguously attributed to the paradigm rather than to benchmark familiarity. The two OOD tasks show divergent outcomes (AA-LCR: +13.3 over the base model; LongBenchV2: ≈0), which makes an explicit breakdown important for interpreting the main claim — and, given the strong AA-LCR result, likely favorable to the authors.

W2 — Missing controls for attribution and for specialization side effects. (a) There is no untrained control on H2S-Bench — e.g., the base model prompted to follow the same Evidence–Summary–Answer format, or an external two-pass summarize-then-answer pipeline — although the motivating study (Table 1) shows training-free summaries already yield large gains, so the net contribution of training beyond the paradigm is unmeasured. (b) No general-capability evaluation accompanies the specialized full-parameter tuning (short-context QA, instruction following, or whether the trained format leaks to non-long-context inputs); peer work such as LoongRL reports such checks. (c) The leading results hold under a 4K output budget; at 16K output Qwen3.8-27B (43.72) exceeds H2S-14B (33.56). This is disclosed (Table 4), but the headline claims are conditional on the tight-budget setting.

Questions for authors

Q1 Please state the example-level deduplication/split protocol between H2S-Dataset and H2S-Bench for the four shared benchmark families (MRCR, Frames, DocFinQA, LongCite), and report Table 2's averages split into in-distribution vs. OOD tasks.

Q2 Section 4.1 states that all models share the same system prompt and block-structured input. Nevertheless, untrained baselines rarely emit the trained block-ID format (e.g., E-Recall of 2.71 for Qwen3.8-27B in Table 5): how should their E-Recall/ESQ entries be interpreted? Reporting the format-parse rate, or an ID-free evidence metric, would help separate format compliance from evidence-finding ability.

Q3 What value of λ is used in Eq. (4), and how sensitive are the H2S-RL gains to this weighting?

Q4 An untrained, prompted H2S-format baseline and/or an external two-pass summarization baseline on H2S-Bench would cleanly separate paradigm gains from training gains.

Q5 A brief comparison of H2S vs. its base model on general capabilities (short-context QA, instruction following) would quantify the cost of specialization.

3 comments

Thank you for your constructive feedback and for recognizing the value of explicit evidence integration, process supervision, and transparent evaluation in H2S.

W1&Q1: Benchmark-family overlap obscures generalization beyond the training distribution.

W1: Train/eval benchmark-family overlap without an ID/OOD breakdown. Four of the seven H2S-Bench tasks (MRCR, Frames, DocFinQA, LongCite) also appear among the 11 H2S-Dataset training families (Figure A6), and Table 2 does not separate in-distribution from OOD performance. As reported, the headline average cannot be unambiguously attributed to the paradigm rather than to benchmark familiarity. The two OOD tasks show divergent outcomes (AA-LCR: +13.3 over the base model; LongBenchV2: ~0), which makes an explicit breakdown important for interpreting the main claim - and, given the strong AA-LCR result, likely favorable to the authors.

Q1: Please state the example-level deduplication/split protocol between H2S-Dataset and H2S-Bench for the four shared benchmark families (MRCR, Frames, DocFinQA, LongCite), and report Table 2's averages split into in-distribution vs. OOD tasks.

Thank you for raising this point. We would like to clarify that the overlap is at the benchmark-family level, while the training and evaluation examples are separated. For the four shared families (DocFinQA, Frames, LongCite, and MRCR), we first divide each source pool into disjoint training and held-out pools, construct H2S-Bench from the held-out examples, and deduplicate documents against the training set using hashing and embedding similarity.

We would also like to clarify that H2S-Bench comprises four ID tasks from these shared families and three OOD tasks (AA-LCR, LongBenchV2, and HELMET-Summ) excluded from training. Following your request, we report their averages separately below.

ModelID-Avg (4 tasks)OOD-Avg (3 tasks)Avg (7 tasks)
Qwen2.5-7B-Instruct-1M7.0812.849.55
Qwen2.5-14B-Instruct-1M18.1420.4719.14
Qwen3.8-27B31.1710.7822.43
QwenLong-L1-32B29.4625.127.59
H2S-7B33.721.7228.56
H2S-14B37.1426.5532.6

H2S-7B improves over its base model by 26.62 points on ID tasks and 8.88 on OOD tasks, the corresponding gains for H2S-14B are 19.00 and 6.08 points. These results show improvements beyond the training benchmark families. As you note, the OOD gains vary across tasks, with a substantial improvement on AA-LCR and little change on LongBenchV2, we therefore interpret the OOD average as an aggregate gain rather than a uniform improvement across tasks.

W2-b&Q5: Effects of specialization on general capabilities and output behavior remain unclear.

W2-b: No general-capability evaluation accompanies the specialized full-parameter tuning (short-context QA, instruction following, or whether the trained format leaks to non-long-context inputs); peer work such as LoongRL reports such checks.

Q5: A brief comparison of H2S vs. its base model on general capabilities (short-context QA, instruction following) would quantify the cost of specialization.

Thank you for raising this question. Our primary objective is to enable efficient answering of long-context questions through evidence extraction and context compression. To additionally assess whether this training preserves general capabilities, we followed your request and evaluated H2S-7B and its base model on MMLU, MATH-500, and IFEval. We randomly sampled 200 examples from each benchmark, totaling 600 examples, and used identical prompts and inputs for both models.

BenchmarkQwen2.5-7B-Instruct-1MH2S-7BΔ
MMLU68.565.5-3
MATH-50071.571-0.5
IFEval76.772.8-3.9
Avg72.2369.77-2.47

H2S-7B largely preserves performance on these short-context tasks, with an average score of 69.77 versus 72.23 for the base model—a modest decrease of 2.47 points (3.4% relative). The individual decreases range from 0.5 to 3.9 points, indicating that the gains in long-context performance are accompanied by limited reductions in the general capabilities evaluated here.

W2-a&Q4: Contributions of the H2S pipeline and training remain insufficiently distinguished.

W2-a: Missing controls for attribution and for specialization side effects. (a) There is no untrained control on H2S-Bench - e.g., the base model prompted to follow the same Evidence-Summary-Answer format, or an external two-pass summarize-then-answer pipeline - although the motivating study (Table 1) shows training-free summaries already yield large gains, so the net contribution of training beyond the paradigm is unmeasured.

Q4: An untrained, prompted H2S-format baseline and/or an external two-pass summarization baseline on H2S-Bench would cleanly separate paradigm gains from training gains.

Thank you for raising this point. All baselines in Table 2 were instructed to follow the same Highlight–Summarize–Answer pipeline. The Qwen2.5-7B/14B-Instruct-1M scores of 9.55 and 19.14 therefore already correspond to the untrained, H2S-prompted controls you suggest. To make the contributions of prompting and training explicit, we include Direct answering, CoT [1], RAG [2], and Self-consistency [3] alongside the prompted and trained H2S models below.

ModelPromptTemplateDocFinQAFramesLongCiteMRCRAA-LCRLongBenchV2HELMET-SummAvg
Qwen2.5-7B-Instruct-1MDirect10.720.755.895.89.5423.813.7612.89
CoT20.920.416.0668.7425.9613.7914.55
RAG17.2315.656.29114.5221.8813.7112.9
Self-consistency20.920.076.045.29.1826.4413.7914.52
Prompted H2S513.955.374.017.3124.047.169.55
H2S-7b-SFTPrompted H2S20.4531.298.6558.1213.9627.885.4223.68
H2S-7bPrompted H2S26.3332.9715.7659.7221.4732.1111.5828.56
Qwen2.5-14B-Instruct-1MDirect7.1229.938.3517.417.427.414.0617.38
CoT17.9828.238.8615.816.6128.6114.0818.6
RAG21.222.117.210.818.6221.3912.8414.88
Self-consistency18.827.58.41517.0529.713.918.62
Prompted H2S15.723.478.3225.0513.5635.8212.0319.14
H2S-14b-SFTPrompted H2S31.4937.7610.565.1322.428.378.4529.16
H2S-14bPrompted H2S30.8337.7213.8866.1326.8635.4217.3832.6

For the 14B model, H2S prompting improves the score from 17.38 to 19.14, showing that the paradigm can be effective without fine-tuning. At 7B, it scores 9.55 versus 12.89 for Direct; our manual inspection suggests that the smaller model has difficulty reliably following the three-stage instructions.

Training the models to follow this response paradigm produces substantially larger gains. Under the same H2S prompt, SFT raises the 7B/14B scores to 23.68/29.16, and SFT+GRPO further raises them to 28.56/32.60. Compared with the untrained H2S-prompted controls, the full training pipeline adds 19.01 and 13.46 points, respectively, demonstrating improvements beyond the prompting paradigm alone.

W2-c: Performance advantages depend on the output budget.

W2-c: The leading results hold under a 4K output budget; at 16K output Qwen3.8-27B (43.72) exceeds H2S-14B (33.56). This is disclosed (Table 4), but the headline claims are conditional on the tight-budget setting

Thank you for raising this point. Our primary objective is to provide high-quality answers to long-context questions with efficient generation. We also evaluated a 16K output budget to examine whether H2S benefits from longer generation. Both models improve: H2S-7B increases from 28.56 to 30.48, and H2S-14B from 32.6 to 33.56. The table below reports task scores, completion rates, and actual output lengths across all 2575 questions, with outputs retokenized using the Qwen2.5 tokenizer.

Model (Budget)Max Output TokenMean output tokensAvgComplete
Qwen3.8-27B4k3,31122.4341.17%
16k6,66043.7286.60%
H2S-7B4k1,45428.5678.14%
16k3,03130.4893.13%
H2S-14B4k1,06532.687.11%
16k2,15433.5693.98%

H2S delivers strong answer quality with substantially shorter outputs. Under the 4K budget, H2S-14B scores 32.6 versus 22.43 for Qwen3.8-27B, while generating only 1,065 tokens on average versus 3,311. Its completion rate is also higher, at 87.11% versus 41.17%. These results illustrate the benefit of extracting and compressing evidence before answering.

Q2: Block-ID-based metrics may conflate format compliance with evidence retrieval quality.

Q2: Section 4.1 states that all models share the same system prompt and block-structured input. Nevertheless, untrained baselines rarely emit the trained block-ID format (e.g., E-Recall of 2.71 for Qwen3.8-27B in Table 5): how should their E-Recall/ESQ entries be interpreted? Reporting the format-parse rate, or an ID-free evidence metric, would help separate format compliance from evidence-finding ability.

Thank you for this question. ER measures exact matches of the complete eight-character block ID emitted within <evidence> against the annotated evidence IDs. Qwen3.8-27B frequently produces malformed or nonexistent IDs, with a Parse-BlockID rate of only 19.8% and an ER of 2.71. Parse-BlockID measures compliance with the required ID format, whereas ER additionally requires the ID to identify a reference evidence block. To separate format compliance from evidence-finding ability, we additionally report Soft-ER. instead of matching emitted block IDs, it compares the text of each predicted claim inside <evidence> with the corresponding annotated claim using ROUGE-L.

CategoryModelParse-BlockIDERSoft-ERDelta
ClosedClaude Opus 510056.3163.296.98
ClosedGPT-5.510041.147.246.14
ClosedDeepSeek-V4-Pro10045.4754.689.21
ClosedKimi-K2.510048.9657.288.32
ClosedGLM-5.210051.7559.567.81
OpenQwen3-4B-Thinking-250782.914.9416.481.54
OpenQwen3.8-27B19.82.7122.5219.81
OpenQwen3.5-35B-A3B78.515.3222.987.66
OpenQwenLong-L1-32B99.834.3141.276.96
OpenQwenLong-L1.5-30B-A3B68.97.7418.510.76
OpenQwen2.5-7B-Instruct-1M10014.4918.43.91
OpenQwen2.5-14B-Instruct-1M99.928.3234.055.73
OursH2S-7B-SFT10036.1244.168.04
OursH2S-7B10038.8547.288.43
OursH2S-14B-SFT10043.0652.769.7
OursH2S-14B10044.9454.299.35

H2S outperforms Qwen3.8-27B under both strict ID matching and soft content matching. H2S-7B/14B achieve ER scores of 38.85/44.94, compared with 2.71 for Qwen3.8-27B, and soft-ER scores of 47.28/54.29, compared with 22.52. Both H2S models also achieve a 100% Parse-BlockID rate. These results show that H2S retrieves relevant evidence more effectively and identifies its source blocks more reliably; its advantage holds whether or not exact block-ID matching is required.

Q3: The reward weight and sensitivity to its choice are unspecified.

Q3: What value of lambda is used in Eq. (4), and how sensitive are the H2S-RL gains to this weighting?

Thank you for this question. We use λ = 0.6 in Equation (4), which assigns weights of 0.6 and 0.4 to the final-answer reward and the process reward, respectively. We will state this value explicitly in the main text.

We selected this weighting during training to balance answer correctness with the quality of the intermediate evidence and summary. When λ was too large, the answer reward dominated and provided less constraint on the intermediate stages. When λ was too small, the process reward dominated before the answer reward had sufficiently converged. Among the settings we explored, λ = 0.6 gave the most stable behavior across both model sizes and the seven tasks, so we used it throughout the H2S-RL experiments. These observations explain our choice; a quantitative sensitivity curve is not included in the current results.

Reviewer_z48fOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes H2S, a compress-then-reason paradigm that forces the model to output evidence (with block IDs) → summary → answer, trained with SFT plus GRPO RL on a 6,647-example dataset (avg 43.9K tokens). H2S-14B scores 32.60 on H2S-Bench, beating Qwen3.8-27B by 10.17 points under a 4K output budget.

Critical strengths
  1. The evidence→summary→answer trace is simple, general, and inspectable. It also gives a natural substrate for future provenance and faithfulness checks.
  2. Strong gains at tiny output budgets: 4K already retains 97.1% of 16K performance. Baselines by contrast nearly double their score only at 16K, showing the compression is doing real work.
  3. Ablations isolate each reward cleanly. Summary and answer rewards contribute the most (−8.48 and −11.65 when removed), while evidence reward still helps (−6.14).
Critical weaknesses
  1. H2S-Bench is assembled from the same 11 benchmark families used to build training data. Without held-out source filtering, in-domain contamination is a real risk.
  2. All baselines are Qwen-family. It is unclear whether the gains transfer to GPT, Llama, or DeepSeek backbones.
  3. No comparison with obvious alternatives such as retrieval-then-read, self-consistency, or chain-of-note. These are the natural competitors for the same budget.
  4. ESQ is judged by a single LLM (Qwen2.5-7B-Instruct-1M) with no human agreement study. Inter-rater reliability is unreported.
Questions for authors
  1. How sensitive are the results to chunk size, top-k retrieval, and the reranker choice?
  2. What is the latency and token overhead of the three-stage output compared to direct answering at matched quality?
  3. Can the claim-span entailment check be made explicit and supervised, rather than prompt-based?
4 comments

We thank you for the thoughtful feedback and for recognizing the simplicity, interpretability, output efficiency, and reward design of H2S.

W1: Shared benchmark sources raise concerns about training–evaluation contamination.

W1: H2S-Bench is assembled from the same 11 benchmark families used to build training data. Without held-out source filtering, in-domain contamination is a real risk.

Thank you for raising this point. We would like to clarify that the overlap is at the benchmark-family level, while the training and evaluation examples are separated.For the four shared benchmark families (DocFinQA, Frames, LongCite, and MRCR), we construct H2S-Bench from held-out examples and deduplicate documents against the training set using hashing and embedding similarity.

We would also like to clarify that H2S-Bench comprises four ID tasks (DocFinQA, Frames, LongCite, and MRCR) with held-out examples from shared benchmark families and three OOD tasks (AA-LCR, LongBenchV2, and HELMET-Summ) excluded from training, as reported separately below.

ModelID-Avg (4 tasks)OOD-Avg (3 tasks)Avg (7 tasks)
Qwen2.5-7B-Instruct-1M7.0812.849.55
Qwen2.5-14B-Instruct-1M18.1420.4719.14
Qwen3.8-27B31.1710.7822.43
QwenLong-L1-32B29.4625.127.59
H2S-7B33.721.7228.56
H2S-14B37.1426.5532.6

Q1: Sensitivity to chunk size, retrieval depth, and reranker choice remains unclear.

Q1: How sensitive are the results to chunk size, top-k retrieval, and the reranker choice?

Thank you for the question. A full sensitivity analysis would require reconstructing the training data and retraining models which we cannot complete within the rebuttal period. We would like to clarify why we selected the current parameters.

We selected the chunk size to balance evidence localization granularity against the input overhead introduced by block-ID markers. Among the five candidate sizes (64, 128, 256, 512, and 1024), smaller chunks incur substantial marker overhead (15.30% and 8.29% for 64 and 128), otherwise larger chunks reduce the resolution of evidence selection. A chunk size of 256 yields an average of 171.5 blocks per document with 4.33% marker overhead, providing fine-grained retrieval units at a moderate input cost. We therefore adopted this setting for subsequent evidence retrieval and training-data construction.

Chunk SizeTotal chunksChunk Num (per document)Marker tokens (per document)Marker proportion
644,549,539684.47,52915.30%
1282,276,432342.53,7678.29%
2561,139,895171.51,8864.33%
512571,612869462.22%
1024287,41343.24761.13%

With the chunk size fixed at 256, we used RRF to combine the embedding and BM25 rankings and selected the retrieval depth according to evidence recall and output length. Our aim was to retain sufficient evidence for training while keeping the generated evidence within the 4K budget. The table below summarizes both criteria for each top-k setting.

top-kEvidence recall (RRF)Mean tokensP95 tokenProportion exceeding 4K
80.7347829360%
160.8571,4061,6570%
320.9432,5963,0041.70%
640.993,7304,89221.50%
1280.9956,6087,55666.50%

Top-32 provides the most suitable balance among the evaluated settings. Increasing k from 8 or 16 to 32 raises recall from 0.734 or 0.857 to 0.943, while the mean and P95 output lengths remain at 2,596 and 3,004 tokens, respectively; only 1.70% of cases exceed 4K. Increasing k further to 64 yields a smaller recall improvement of 0.047, but raises the P95 length to 4,892 tokens and the proportion exceeding 4K to 21.50%. At k=128, recall improves only to 0.995, while 66.50% of cases exceed the budget.

Following this selection procedure, we adopted a chunk size of 256 and a retrieval depth of 32.

W2: Generalization beyond Qwen backbones remains uncertain.

W2: All baselines are Qwen-family. It is unclear whether the gains transfer to GPT, Llama, or DeepSeek backbones.

Thank you for this suggestion. We additionally evaluated H2S on GPT-6.1-sol, DeepSeek-V4-Pro, and Kimi-K2.5. Given their scale and the computational cost of full-parameter training, we compared H2S prompting with Direct answering on four tasks under identical input and evaluation settings.

ModelPromptTemplateAA-LCRDocFinQAFramesLongBenchV2Avg
GPT-6.1-solDirect39.8550.6350.468.852.42
GPT-6.1-solH2S51.0850.554870.455.01
DeepSeek-V4-ProDirect45.436.0934.5745.6740.43
DeepSeek-V4-ProH2S47.7734.8337.4148.5642.14
Kimi-K2.5Direct36.3751.8734.4546.8542.39
Kimi-K2.5H2S39.9551.1437.415044.63

H2S improves average performance by 2.59, 1.71, and 2.24 points, respectively, demonstrating that the prompting paradigm benefits non-Qwen backbones without additional training.

Q2: Latency and token overhead at matched answer quality remain unclear.

Q2: What is the latency and token overhead of the three-stage output compared to direct answering at matched quality?

Thank you for this question. In five cases where both methods answer correctly, H2S adds 122.4 output tokens and 8.06 seconds of end-to-end latency on average. We use the same model (Qwen2.5-7B-Instruct-1M), 128K input and 4K output budgets, and inference configuration. Specifically, we select one example each from MRCR, DocFinQA, LongBenchV2, AA-LCR, and Frames for which both Direct and H2S produce correct answers, then measure token usage, time to first token (TTFT), and end-to-end latency. Mean denotes the arithmetic mean over these five cases.

MetricPromptTemplateMRCRDocFinQALongBenchV2AA-LCRFramesMean
Input tokenDirect7,816118,46319,708121,3992,73854,024.80
H2S7,898118,54519,790121,4212,82054,094.80
Output tokenDirect723625411237.2
H2S21118281192132159.6
Evidence529537908471.6
TTFT (s)Direct4.32161.9715.84166.851.2870.05
H2S4.28169.615.78168.421.2871.87
Latency (s)Direct6.73163.416.65168.411.6571.37
H2S20.61176.5918.46175.825.6579.43

The additional token cost mainly comes from making the intermediate evidence and summary explicit. Input lengths remain nearly identical, while the mean output increases from 37.2 tokens for Direct to 159.6 for H2S, a difference of 122.4 tokens. Of the H2S output, 71.6 tokens are used for evidence on average, making this stage a substantial part of the additional generation cost.

The corresponding latency increase is 1.82 seconds for TTFT (71.87s vs. 70.05s, approximately 2.6%) and 8.06 seconds end to end (79.43s vs. 71.37s, approximately 11.3%). We selected these cases because both methods answered correctly, allowing the overhead to be examined at matched answer correctness. We include the individual measurements as well as the mean to make the variation visible; these five cases illustrate the cost rather than establish a benchmark-wide latency estimate.

W3: Comparisons with alternative retrieval and reasoning methods are incomplete.

W3: No comparison with obvious alternatives such as retrieval-then-read, self-consistency, or chain-of-note. These are the natural competitors for the same budget.

Thank you for pointing out this ambiguity. All baselines in Table 2 were instructed to follow the same Highlight–Summarize–Answer pipeline. This means that the Qwen2.5-7B/14B-Instruct-1M scores of 9.55 and 19.14 already correspond to the untrained, H2S-prompted baselines you suggest. We agree that this setup should have been stated more clearly. To make the comparison explicit, we include Direct answering and other prompting baselines in the table below.

ModelPromptTemplateDocFinQAFramesLongCiteMRCRAA-LCRLongBenchV2HELMET-SummAvg
Qwen2.5-7B-Instruct-1MDirect10.720.755.895.89.5423.813.7612.89
CoT20.920.416.0668.7425.9613.7914.55
RAG17.2315.656.29114.5221.8813.7112.9
Self-consistency20.920.076.045.29.1826.4413.7914.52
Prompted H2S513.955.374.017.3124.047.169.55
H2S-7b-SFTPrompted H2S20.4531.298.6558.1213.9627.885.4223.68
H2S-7bPrompted H2S26.3332.9715.7659.7221.4732.1111.5828.56
Qwen2.5-14B-Instruct-1MDirect7.1229.938.3517.417.427.414.0617.38
CoT17.9828.238.8615.816.6128.6114.0818.6
RAG21.222.117.210.818.6221.3912.8414.88
Self-consistency18.827.58.41517.0529.713.918.62
Prompted H2S15.723.478.3225.0513.5635.8212.0319.14
H2S-14b-SFTPrompted H2S31.4937.7610.565.1322.428.378.4529.16
H2S-14bPrompted H2S30.8337.7213.8866.1326.8635.4217.3832.6

The results show why the distinction between prompting and training matters. For the 7B model, H2S prompting alone scores 9.55, below Direct answering at 12.89; for the 14B model, it improves the score from 17.38 to 19.14. Our manual inspection suggests that the 7B model has difficulty reliably following the three-stage instructions. The improvement at 14B shows that H2S prompting can already be effective, and training the model to follow this response paradigm leads to substantially better performance, as shown below.

References:

[1] Wei et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS. (CoT)

[2] Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. (RAG)

[3] Wang et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR. (Self-consistency)

Q3: Claim–span entailment relies on prompt-based checks rather than explicit supervision.

Q3: Can the claim-span entailment check be made explicit and supervised, rather than prompt-based?

Thank you for this question. Our data-construction pipeline is LLM-based because it requires generating sub-queries and formulating their answers as claims grounded in the document. These steps involve question decomposition, semantic interpretation, and abstraction, which are not provided by retrieval scores or string matching alone. In our pipeline, the LLM therefore serves an essential role in constructing the sub-query–claim–span associations, beyond checking already available pairs.

Specifically, we retrieve candidate blocks using embeddings and BM25 and combine their rankings with RRF. The LLM then decomposes the question into sub-queries, identifies the relevant source spans, and formulates a claim answering each sub-query from those spans. The resulting claims and sub-queries are used to construct the summary, while the explicit claim–span associations retain the supporting evidence for validation.

We assess these associations by comparing matched claim–span pairs with mismatched pairs formed from similar spans in the same document. Matched pairs achieve an average embedding cosine similarity of 66.3, compared with 40.6 for mismatched pairs, with an average AUC of 85.5. This separation provides quantitative support that the extracted claims align with their supporting spans and can be distinguished from alternative spans within the same document.

MetricAA-LCRDocFinQAFramesHELMET-SummLongBenchV2LongCiteMRCRAVG
Positive-pair cos61.764.8695668.477.167.166.3
Negative-pair cos45.239.638.530.543.847.339.240.6
AUC73.285.388.189.586.591.184.385.5

Here, cos denotes embedding cosine similarity; positive and negative pairs are matched and mismatched claim–span pairs, respectively. AUC measures how reliably matched pairs score above mismatched pairs and is reported on a 0–100 scale. These metrics assess matching consistency; complete evidence coverage and absence of duplicate claims require separate coverage and redundancy measurements.

W4: Reliance on a single LLM judge raises concerns about ESQ reliability.

W4: ESQ is judged by a single LLM (Qwen2.5-7B-Instruct-1M) with no human agreement study. Inter-rater reliability is unreported.

We would like to clarify that ESQ is computed programmatically and does not use an LLM judge. We calculate ESQ from two reproducible automatic metrics. E-Recall measures evidence-block recall by matching predicted eight-character block IDs against annotated IDs. S-RL measures summary similarity using ROUGE-L between the predicted and reference summaries. Neither component requires a subjective judgment from an LLM. ESQ=100N∑i=1N2 ERi SRLiERi+SRLi\mathrm{ESQ}=\frac{100}{N}\sum_{i=1}^{N}\frac{2\,\mathrm{ER}_i\,\mathrm{SRL}_i}{\mathrm{ER}_i+\mathrm{SRL}_i}

Thus, the ESQ score does not depend on subjective judgments made by a single LLM. Inter-rater agreement is not part of this deterministic calculation, although the reliability of the reference annotations is a separate question. We will make this distinction between reference construction and metric computation explicit in the manuscript.

Reviewer_HFfNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents the Highlight-Then-Summarize (H2S) framework for improving LLM’s long-context understanding by finding (highlighting) relevant text snippets (evidence) and then summarizing them into concise question-conditioned representation. In addition, the contributions include H2S-Dataset, consisting of 6647 samples (4228 samples for SFT + 2419 samples for RL) from 11 benchmark families, and H2S-RL method that uses a reward that allows to evaluate the final answer quality as well as “evidence grounding and span coverage, summary quality, and output structure”. The authors evaluate the proposed H2S and open/closed-source models on long-context benchmarks, achieving the highest average score.

Critical strengths
  1. The main strength is the construction of H2S-Dataset, with evidence annotations and question-conditioned summaries. This dataset could support further research and be useful to the research community.

  2. The paper is clearly written and easy to follow.

Critical weaknesses

1.The comparison with open- and closed-source models needs clarification (Table 2). It is unclear whether the baselines answered directly from the context or were explicitly instructed to follow the Highlight–Summarize–Answer pipeline. Could prompting them to extract evidence, summarize it, and use that summary improve their performance without fine-tuning? Re-evaluating every baseline is unnecessary. A relevant comparison would evaluate Qwen2.5-7B/14B-Instruct-1M with this prompt before fine-tuning and compare them with the fine-tuned models.

2.The training ablations do not sufficiently separate the effect of GRPO from additional data and computation. The reported splits contain 4,228 SFT examples and 2,419 additional RL examples, giving 6,647 examples for the combined SFT + GRPO pipeline (Section 3.3; Appendix A.2). GRPO also adds training computation after SFT. This puts the SFT-only baseline in Table 2 at a disadvantage. The comparison should include SFT on all collected examples, matching the data and computation available to the combined pipeline.

3.The GRPO-specific contribution is the proposed reward function (Section 3.4, Equation 4). However, its benefit is not directly tested. The authors should compare GRPO using this reward with GRPO using only final-answer quality as the reward. This would help establish the value of the proposed evidence and summary reward components.

Questions for authors
  • Were the baselines in Table 2 prompted to answer directly from the context or to follow the Highlight–Summarize–Answer pipeline? Could you evaluate Qwen2.5-7B/14B-Instruct-1M with this pipeline prompt, without fine-tuning, and compare them with the fine-tuned models?

  • Could you add an SFT-only baseline trained on all 6,647 examples available to the combined SFT + GRPO pipeline?

  • Could you compare GRPO using the proposed reward (Section 3.4, Equation 4) with GRPO using a reward based only on final-answer quality?

2 comments

We thank you for the constructive feedback and for recognizing the value of H2S-Dataset and its potential to support further research in long-context understanding.

W1&Q1: Unclear baseline prompts obscure the respective contributions of prompting and training.

W1: The comparison with open- and closed-source models needs clarification (Table 2). It is unclear whether the baselines answered directly from the context or were explicitly instructed to follow the Highlight-Summarize-Answer pipeline. Could prompting them to extract evidence, summarize it, and use that summary improve their performance without fine-tuning? Re-evaluating every baseline is unnecessary. A relevant comparison would evaluate Qwen2.5-7B/14B-Instruct-1M with this prompt before fine-tuning and compare them with the fine-tuned models.

Q1: Were the baselines in Table 2 prompted to answer directly from the context or to follow the Highlight-Summarize-Answer pipeline? Could you evaluate Qwen2.5-7B/14B-Instruct-1M with this pipeline prompt, without fine-tuning, and compare them with the fine-tuned models?

Thank you for pointing out this ambiguity. All baselines in Table 2 were instructed to follow the same Highlight–Summarize–Answer pipeline. This means that the Qwen2.5-7B/14B-Instruct-1M scores of 9.55 and 19.14 already correspond to the untrained, H2S-prompted baselines you suggest. We agree that this setup should have been stated more clearly. To make the comparison explicit, we include Direct answering and other prompting baselines in the table below.

ModelPromptTemplateDocFinQAFramesLongCiteMRCRAA-LCRLongBenchV2HELMET-SummAvg
Qwen2.5-7B-Instruct-1MDirect10.720.755.895.89.5423.813.7612.89
CoT20.920.416.0668.7425.9613.7914.55
RAG17.2315.656.29114.5221.8813.7112.9
Self-consistency20.920.076.045.29.1826.4413.7914.52
Prompted H2S513.955.374.017.3124.047.169.55
H2S-7b-SFTPrompted H2S20.4531.298.6558.1213.9627.885.4223.68
H2S-7bPrompted H2S26.3332.9715.7659.7221.4732.1111.5828.56
Qwen2.5-14B-Instruct-1MDirect7.1229.938.3517.417.427.414.0617.38
CoT17.9828.238.8615.816.6128.6114.0818.6
RAG21.222.117.210.818.6221.3912.8414.88
Self-consistency18.827.58.41517.0529.713.918.62
Prompted H2S15.723.478.3225.0513.5635.8212.0319.14
H2S-14b-SFTPrompted H2S31.4937.7610.565.1322.428.378.4529.16
H2S-14bPrompted H2S30.8337.7213.8866.1326.8635.4217.3832.6

The results show why the distinction between prompting and training matters. For the 7B model, H2S prompting alone scores 9.55, below Direct answering at 12.89; for the 14B model, it improves the score from 17.38 to 19.14. Our manual inspection suggests that the 7B model has difficulty reliably following the three-stage instructions. The improvement at 14B shows that H2S prompting can already be effective, and training the model to follow this response paradigm leads to substantially better performance, as shown below.

References:

[1] Wei et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS. (CoT)

[2] Lewis et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS. (RAG)

[3] Wang et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR. (Self-consistency)

W2&Q2: Additional data and computation confound the gains attributed to GRPO.

W2: The training ablations do not sufficiently separate the effect of GRPO from additional data and computation. The reported splits contain 4,228 SFT examples and 2,419 additional RL examples, giving 6,647 examples for the combined SFT + GRPO pipeline (Section 3.3; Appendix A.2). GRPO also adds training computation after SFT. This puts the SFT-only baseline in Table 2 at a disadvantage. The comparison should include SFT on all collected examples, matching the data and computation available to the combined pipeline.

Q2: Could you add an SFT-only baseline trained on all 6,647 examples available to the combined SFT + GRPO pipeline?

Thank you for this suggestion. We would like to clarify that the original SFT stage was designed to establish the basic ability to follow the H2S pipeline, and the 4,228 examples already yield substantial improvements over the base models. We also expect that adding more examples and more diverse supervision can strengthen this foundation. Following your suggestion, we therefore combined the 4,228 SFT examples with the 2,419 examples originally reserved for RL and trained SFT-only models on all 6,647 examples, keeping the original SFT hyperparameters. We refer to these models as H2S-7B-SFT-Full and H2S-14B-SFT-Full.

H2S continues to outperform SFT-only training when both pipelines have access to all 6,647 examples, by 1.45 points at 7B and 1.55 points at 14B. Adding the RL examples to SFT raises its scores from 23.68 to 27.11 (+3.43) and from 29.16 to 31.05 (+1.89), respectively, while H2S achieves 28.56 and 32.60. The advantage is also evident on HELMET-Summ: H2S scores 11.58/17.38, compared with 6.84/12.21 for SFT-Full.

ModelDocFinQAFramesLongCiteMRCRAA-LCRLongBenchV2HELMET-SummAvg
Qwen2.5-7B-Instruct-1M513.955.374.017.3124.047.169.55
Qwen2.5-14B-Instruct-1M15.723.478.3225.0513.5635.8212.0319.14
H2S-7B-SFT20.4531.298.6558.1213.9627.885.4223.68
H2S-7B-SFT-Full32.633.3312.3957.819.1627.646.8427.11
H2S-7B26.3332.9715.7659.7221.4732.1111.5828.56
H2S-14B-SFT31.4937.7610.565.1322.428.378.4529.16
H2S-14B-SFT-Full37.5938.7813.356916.3630.0512.2131.05
H2S-14B30.8337.7213.8866.1326.8635.4217.3832.6

W3&Q3: Benefits of evidence and summary rewards beyond answer-only supervision remain unclear.

W3: The GRPO-specific contribution is the proposed reward function (Section 3.4, Equation 4). However, its benefit is not directly tested. The authors should compare GRPO using this reward with GRPO using only final-answer quality as the reward. This would help establish the value of the proposed evidence and summary reward components.

Q3: Could you compare GRPO using the proposed reward (Section 3.4, Equation 4) with GRPO using a reward based only on final-answer quality?

We agree that an answer-only GRPO control is needed to test whether the evidence and summary rewards add value. We have now added this comparison. At each model size, both variants start from the same H2S-SFT checkpoint and follow the same training procedure as H2S. We change only the reward function: the control uses a dense reward based solely on final-answer correctness in the table.

With this control in place, the contribution of the process rewards becomes clearer. At 7B, answer-only GRPO scores 27.33, while the full H2S reward reaches 28.56, an additional 1.23 points. At 14B, the score increases from 30.62 to 32.60, an additional 1.98 points. These consistent gains support the joint benefit of the evidence and summary rewards beyond final-answer supervision.

SizeModelH2S-BenchΔ vs Baseline
7BQwen2.5-7B-Instruct-1M (baseline)9.550
7BH2S-7B-SFT23.6814.13
7BH2S-7B-GRPO27.3317.78
7BH2S-7B28.5619.01
14BQwen2.5-14B-Instruct-1M (baseline)19.140
14BH2S-14B-SFT29.1610.02
14BH2S-14B-GRPO30.6211.48
14BH2S-14B32.613.46

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.