- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies context compression for multi-hop question answering. It argues that existing compressors often remove intermediate bridge entities even when the final answer evidence remains in the compressed context. The paper introduces a “dual-paradigm theory” consisting of a quality paradigm, which decomposes compression into selection, generation, and arrangement, and an overhead paradigm, which accounts for compressor, reader, auxiliary, and training costs. Based on this framework, the authors propose ChainPack, a two-stage fine-tuning procedure that first trains a model to produce hop-aligned summaries and then performs survival-filtered self-training by retaining generated summaries that contain all annotated bridge entities.
Critical strengths
- The paper addresses a practically relevant failure mode of context compression for multi-hop QA.
- The paper reports token consumptions of the methods.
Critical weaknesses
-
The paper is unnecessarily difficult to follow. The introduction makes broad claims about extractive, abstractive, and iterative methods before providing definitions or experimental support. It introduces the “dual-paradigm theory,” “product law,” and several failure mechanisms in a dense paragraph, while their precise meanings only become clear in Section 3. The claims for weakness of the existing methods also lack for sufficient supports.
-
The theoretical contribution is weak. Most results follow directly from strong assumptions or definitions. Indeed, the theories seem very weird together with their proofs. I doubt the validity and the necessity of such theories.
-
The technical novelty is limited. ChainPack is essentially chain-formatted LoRA fine-tuning followed by rejection-sampling self-training: generate several candidates, retain those containing all gold bridge strings and passing an embedding-similarity threshold, and fine-tune again. This is a reasonable recipe, but it introduces neither a new architecture nor a new optimization objective.
-
The expermental evaluation is rather weak. The main comparison is largely restricted to the LLMLingua family. Direct comparisons with relevant trained, abstractive, or context-aware compressors are missing. Finally, all three evaluation datasets are Wikipedia-style multi-hop QA benchmarks, two of which also supply training data. There is little evidence that the approach generalizes to other RAG settings, tasks without bridge annotations, or even single-hop and out-of-domain QA.
Questions for authors
See the weaknesses.