- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Highlight-Then-Summarize (H2S), a compress-then-reason paradigm for long-context understanding: a single autoregressive model first emits source-grounded evidence (block IDs plus supporting spans) from a block-structured document, then a question-conditioned summary that integrates the selected evidence, and finally the answer. To supervise both intermediate stages, the authors construct H2S-Dataset (6,647 examples from 11 benchmark families, retro-annotated with evidence and summaries via semantic blocking, hybrid retrieval, question decomposition, claim extraction, and faithfulness/coverage/answerability quality checks) and H2S-RL, a GRPO objective with programmatic process rewards combining answer correctness, evidence grounding and span-level F1, summary ROUGE-L, and a structure gate, aggregated via a harmonic mean. Trained from Qwen2.5-7B/14B-Instruct-1M with a 32K→64K→128K curriculum, H2S-14B attains the best average (32.60) among the evaluated open-source models on H2S-Bench (seven tasks, 128K input / 4K output). Process-level analyses (the ESQ metric, component ablations, output-budget studies) support the intended mechanism: explicit compression substitutes for long generation, retaining 97.1% of the 16K-budget performance under a 4K budget.
Critical strengths
S1 — Supervising a genuinely under-supervised stage: evidence integration. As the paper's related-work discussion acknowledges, existing long-context RL methods reward evidence selection or final answers (e.g., LongRLVR's context reward targets selection; LoongRL's reasoning pattern emerges from outcome rewards), leaving the integration of dispersed evidence implicit in answer generation. H2S externalizes this stage as an observable intermediate representation and supervises it with process-level rewards whose harmonic-mean design prevents either component from masking the other. The ablations support the design: removing the summary costs 8.5–9.9 points and removing evidence 4.0–4.3 (Table 3).
S2 — Mechanism-transparent evaluation and honest reporting. The paper evaluates intermediate quality (ESQ, Table 5), isolates where gains originate (output-budget analysis: thinking baselines improve by 11–25 points when allowed 16K output while H2S improves by ≤1, Table 4), and analyzes failure cases at the stage level, distinguishing evidence-selection errors from arithmetic errors (Appendix A.6/A.9). The reported numbers are internally consistent across the abstract, main tables, and appendices, and limitations are stated explicitly.
Critical weaknesses
W1 — Train/eval benchmark-family overlap without an ID/OOD breakdown. Four of the seven H2S-Bench tasks (MRCR, Frames, DocFinQA, LongCite) also appear among the 11 H2S-Dataset training families (Figure A6), and Table 2 does not separate in-distribution from OOD performance. As reported, the headline average cannot be unambiguously attributed to the paradigm rather than to benchmark familiarity. The two OOD tasks show divergent outcomes (AA-LCR: +13.3 over the base model; LongBenchV2: ≈0), which makes an explicit breakdown important for interpreting the main claim — and, given the strong AA-LCR result, likely favorable to the authors.
W2 — Missing controls for attribution and for specialization side effects. (a) There is no untrained control on H2S-Bench — e.g., the base model prompted to follow the same Evidence–Summary–Answer format, or an external two-pass summarize-then-answer pipeline — although the motivating study (Table 1) shows training-free summaries already yield large gains, so the net contribution of training beyond the paradigm is unmeasured. (b) No general-capability evaluation accompanies the specialized full-parameter tuning (short-context QA, instruction following, or whether the trained format leaks to non-long-context inputs); peer work such as LoongRL reports such checks. (c) The leading results hold under a 4K output budget; at 16K output Qwen3.8-27B (43.72) exceeds H2S-14B (33.56). This is disclosed (Table 4), but the headline claims are conditional on the tight-budget setting.
Questions for authors
Q1 Please state the example-level deduplication/split protocol between H2S-Dataset and H2S-Bench for the four shared benchmark families (MRCR, Frames, DocFinQA, LongCite), and report Table 2's averages split into in-distribution vs. OOD tasks.
Q2 Section 4.1 states that all models share the same system prompt and block-structured input. Nevertheless, untrained baselines rarely emit the trained block-ID format (e.g., E-Recall of 2.71 for Qwen3.8-27B in Table 5): how should their E-Recall/ESQ entries be interpreted? Reporting the format-parse rate, or an ID-free evidence metric, would help separate format compliance from evidence-finding ability.
Q3 What value of λ is used in Eq. (4), and how sensitive are the H2S-RL gains to this weighting?
Q4 An untrained, prompted H2S-format baseline and/or an external two-pass summarization baseline on H2S-Bench would cleanly separate paradigm gains from training gains.
Q5 A brief comparison of H2S vs. its base model on general capabilities (short-context QA, instruction following) would quantify the cost of specialization.
3 comments
W2-a&Q4: Contributions of the H2S pipeline and training remain insufficiently distinguished.
W2-a: Missing controls for attribution and for specialization side effects. (a) There is no untrained control on H2S-Bench - e.g., the base model prompted to follow the same Evidence-Summary-Answer format, or an external two-pass summarize-then-answer pipeline - although the motivating study (Table 1) shows training-free summaries already yield large gains, so the net contribution of training beyond the paradigm is unmeasured.
Q4: An untrained, prompted H2S-format baseline and/or an external two-pass summarization baseline on H2S-Bench would cleanly separate paradigm gains from training gains.
Thank you for raising this point. All baselines in Table 2 were instructed to follow the same Highlight–Summarize–Answer pipeline. The Qwen2.5-7B/14B-Instruct-1M scores of 9.55 and 19.14 therefore already correspond to the untrained, H2S-prompted controls you suggest. To make the contributions of prompting and training explicit, we include Direct answering, CoT [1], RAG [2], and Self-consistency [3] alongside the prompted and trained H2S models below.
| Model | PromptTemplate | DocFinQA | Frames | LongCite | MRCR | AA-LCR | LongBenchV2 | HELMET-Summ | Avg |
|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-7B-Instruct-1M | Direct | 10.7 | 20.75 | 5.89 | 5.8 | 9.54 | 23.8 | 13.76 | 12.89 |
| CoT | 20.9 | 20.41 | 6.06 | 6 | 8.74 | 25.96 | 13.79 | 14.55 | |
| RAG | 17.23 | 15.65 | 6.29 | 1 | 14.52 | 21.88 | 13.71 | 12.9 | |
| Self-consistency | 20.9 | 20.07 | 6.04 | 5.2 | 9.18 | 26.44 | 13.79 | 14.52 | |
| Prompted H2S | 5 | 13.95 | 5.37 | 4.01 | 7.31 | 24.04 | 7.16 | 9.55 | |
| H2S-7b-SFT | Prompted H2S | 20.45 | 31.29 | 8.65 | 58.12 | 13.96 | 27.88 | 5.42 | 23.68 |
| H2S-7b | Prompted H2S | 26.33 | 32.97 | 15.76 | 59.72 | 21.47 | 32.11 | 11.58 | 28.56 |
| Qwen2.5-14B-Instruct-1M | Direct | 7.12 | 29.93 | 8.35 | 17.4 | 17.4 | 27.4 | 14.06 | 17.38 |
| CoT | 17.98 | 28.23 | 8.86 | 15.8 | 16.61 | 28.61 | 14.08 | 18.6 | |
| RAG | 21.2 | 22.11 | 7.21 | 0.8 | 18.62 | 21.39 | 12.84 | 14.88 | |
| Self-consistency | 18.8 | 27.5 | 8.4 | 15 | 17.05 | 29.7 | 13.9 | 18.62 | |
| Prompted H2S | 15.7 | 23.47 | 8.32 | 25.05 | 13.56 | 35.82 | 12.03 | 19.14 | |
| H2S-14b-SFT | Prompted H2S | 31.49 | 37.76 | 10.5 | 65.13 | 22.4 | 28.37 | 8.45 | 29.16 |
| H2S-14b | Prompted H2S | 30.83 | 37.72 | 13.88 | 66.13 | 26.86 | 35.42 | 17.38 | 32.6 |
For the 14B model, H2S prompting improves the score from 17.38 to 19.14, showing that the paradigm can be effective without fine-tuning. At 7B, it scores 9.55 versus 12.89 for Direct; our manual inspection suggests that the smaller model has difficulty reliably following the three-stage instructions.
Training the models to follow this response paradigm produces substantially larger gains. Under the same H2S prompt, SFT raises the 7B/14B scores to 23.68/29.16, and SFT+GRPO further raises them to 28.56/32.60. Compared with the untrained H2S-prompted controls, the full training pipeline adds 19.01 and 13.46 points, respectively, demonstrating improvements beyond the prompting paradigm alone.
W2-c: Performance advantages depend on the output budget.
W2-c: The leading results hold under a 4K output budget; at 16K output Qwen3.8-27B (43.72) exceeds H2S-14B (33.56). This is disclosed (Table 4), but the headline claims are conditional on the tight-budget setting
Thank you for raising this point. Our primary objective is to provide high-quality answers to long-context questions with efficient generation. We also evaluated a 16K output budget to examine whether H2S benefits from longer generation. Both models improve: H2S-7B increases from 28.56 to 30.48, and H2S-14B from 32.6 to 33.56. The table below reports task scores, completion rates, and actual output lengths across all 2575 questions, with outputs retokenized using the Qwen2.5 tokenizer.
| Model (Budget) | Max Output Token | Mean output tokens | Avg | Complete |
|---|---|---|---|---|
| Qwen3.8-27B | 4k | 3,311 | 22.43 | 41.17% |
| 16k | 6,660 | 43.72 | 86.60% | |
| H2S-7B | 4k | 1,454 | 28.56 | 78.14% |
| 16k | 3,031 | 30.48 | 93.13% | |
| H2S-14B | 4k | 1,065 | 32.6 | 87.11% |
| 16k | 2,154 | 33.56 | 93.98% |
H2S delivers strong answer quality with substantially shorter outputs. Under the 4K budget, H2S-14B scores 32.6 versus 22.43 for Qwen3.8-27B, while generating only 1,065 tokens on average versus 3,311. Its completion rate is also higher, at 87.11% versus 41.17%. These results illustrate the benefit of extracting and compressing evidence before answering.
Q2: Block-ID-based metrics may conflate format compliance with evidence retrieval quality.
Q2: Section 4.1 states that all models share the same system prompt and block-structured input. Nevertheless, untrained baselines rarely emit the trained block-ID format (e.g., E-Recall of 2.71 for Qwen3.8-27B in Table 5): how should their E-Recall/ESQ entries be interpreted? Reporting the format-parse rate, or an ID-free evidence metric, would help separate format compliance from evidence-finding ability.
Thank you for this question. ER measures exact matches of the complete eight-character block ID emitted within <evidence> against the annotated evidence IDs. Qwen3.8-27B frequently produces malformed or nonexistent IDs, with a Parse-BlockID rate of only 19.8% and an ER of 2.71. Parse-BlockID measures compliance with the required ID format, whereas ER additionally requires the ID to identify a reference evidence block. To separate format compliance from evidence-finding ability, we additionally report Soft-ER. instead of matching emitted block IDs, it compares the text of each predicted claim inside <evidence> with the corresponding annotated claim using ROUGE-L.
| Category | Model | Parse-BlockID | ER | Soft-ER | Delta |
|---|---|---|---|---|---|
| Closed | Claude Opus 5 | 100 | 56.31 | 63.29 | 6.98 |
| Closed | GPT-5.5 | 100 | 41.1 | 47.24 | 6.14 |
| Closed | DeepSeek-V4-Pro | 100 | 45.47 | 54.68 | 9.21 |
| Closed | Kimi-K2.5 | 100 | 48.96 | 57.28 | 8.32 |
| Closed | GLM-5.2 | 100 | 51.75 | 59.56 | 7.81 |
| Open | Qwen3-4B-Thinking-2507 | 82.9 | 14.94 | 16.48 | 1.54 |
| Open | Qwen3.8-27B | 19.8 | 2.71 | 22.52 | 19.81 |
| Open | Qwen3.5-35B-A3B | 78.5 | 15.32 | 22.98 | 7.66 |
| Open | QwenLong-L1-32B | 99.8 | 34.31 | 41.27 | 6.96 |
| Open | QwenLong-L1.5-30B-A3B | 68.9 | 7.74 | 18.5 | 10.76 |
| Open | Qwen2.5-7B-Instruct-1M | 100 | 14.49 | 18.4 | 3.91 |
| Open | Qwen2.5-14B-Instruct-1M | 99.9 | 28.32 | 34.05 | 5.73 |
| Ours | H2S-7B-SFT | 100 | 36.12 | 44.16 | 8.04 |
| Ours | H2S-7B | 100 | 38.85 | 47.28 | 8.43 |
| Ours | H2S-14B-SFT | 100 | 43.06 | 52.76 | 9.7 |
| Ours | H2S-14B | 100 | 44.94 | 54.29 | 9.35 |
H2S outperforms Qwen3.8-27B under both strict ID matching and soft content matching. H2S-7B/14B achieve ER scores of 38.85/44.94, compared with 2.71 for Qwen3.8-27B, and soft-ER scores of 47.28/54.29, compared with 22.52. Both H2S models also achieve a 100% Parse-BlockID rate. These results show that H2S retrieves relevant evidence more effectively and identifies its source blocks more reliably; its advantage holds whether or not exact block-ID matching is required.
Q3: The reward weight and sensitivity to its choice are unspecified.
Q3: What value of lambda is used in Eq. (4), and how sensitive are the H2S-RL gains to this weighting?
Thank you for this question. We use λ = 0.6 in Equation (4), which assigns weights of 0.6 and 0.4 to the final-answer reward and the process reward, respectively. We will state this value explicitly in the main text.
We selected this weighting during training to balance answer correctness with the quality of the intermediate evidence and summary. When λ was too large, the answer reward dominated and provided less constraint on the intermediate stages. When λ was too small, the process reward dominated before the answer reward had sufficiently converged. Among the settings we explored, λ = 0.6 gave the most stable behavior across both model sizes and the seven tasks, so we used it throughout the H2S-RL experiments. These observations explain our choice; a quantitative sensitivity curve is not included in the current results.
Thank you for your constructive feedback and for recognizing the value of explicit evidence integration, process supervision, and transparent evaluation in H2S.
W1&Q1: Benchmark-family overlap obscures generalization beyond the training distribution.
Thank you for raising this point. We would like to clarify that the overlap is at the benchmark-family level, while the training and evaluation examples are separated. For the four shared families (DocFinQA, Frames, LongCite, and MRCR), we first divide each source pool into disjoint training and held-out pools, construct H2S-Bench from the held-out examples, and deduplicate documents against the training set using hashing and embedding similarity.
We would also like to clarify that H2S-Bench comprises four ID tasks from these shared families and three OOD tasks (AA-LCR, LongBenchV2, and HELMET-Summ) excluded from training. Following your request, we report their averages separately below.
H2S-7B improves over its base model by 26.62 points on ID tasks and 8.88 on OOD tasks, the corresponding gains for H2S-14B are 19.00 and 6.08 points. These results show improvements beyond the training benchmark families. As you note, the OOD gains vary across tasks, with a substantial improvement on AA-LCR and little change on LongBenchV2, we therefore interpret the OOD average as an aggregate gain rather than a uniform improvement across tasks.
W2-b&Q5: Effects of specialization on general capabilities and output behavior remain unclear.
Thank you for raising this question. Our primary objective is to enable efficient answering of long-context questions through evidence extraction and context compression. To additionally assess whether this training preserves general capabilities, we followed your request and evaluated H2S-7B and its base model on MMLU, MATH-500, and IFEval. We randomly sampled 200 examples from each benchmark, totaling 600 examples, and used identical prompts and inputs for both models.
H2S-7B largely preserves performance on these short-context tasks, with an average score of 69.77 versus 72.23 for the base model—a modest decrease of 2.47 points (3.4% relative). The individual decreases range from 0.5 to 3.9 points, indicating that the gains in long-context performance are accompanied by limited reductions in the general capabilities evaluated here.