- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies how context ordering affects evidence retention in sequential likelihood-based compression. It identifies information preemption, where earlier overlapping contexts reduce the scores of later supporting evidence. MORSE combines a reverse-query-likelihood ordering with random permutations and selects among their compressed outputs. Experiments on HotpotQA, 2WikiMultiHopQA, and MuSiQue show improved evidence retention, with less consistent downstream QA gains.
Critical strengths
-
The controlled pair swaps and separation of compression order from presentation order provide useful evidence that ordering affects which facts survive compression. This makes the analysis informative beyond the proposed method.
-
The evaluation includes matched-candidate random search, selection-objective ablations, and multiple scoring models and compression procedures. These controls help substantiate the evidence-retention improvements.
Critical weaknesses
-
Generalization beyond the main settings remains uncertain. Most additional evaluations use 100 examples, and the advantage over RandomSearch-5 is statistically inconclusive under the joint scorer-dataset shift and many context-scaling settings. MuSiQue’s paragraph-level retention metric also does not establish that the necessary facts survive.
-
The practical quality-cost tradeoff is not fully established. Table 2 reports 1.89 seconds for MORSE-5 versus 0.21 seconds for Reverse, despite candidate parallelism across five GPUs. Given the mixed downstream QA improvements, comparisons with established compression systems at matched total cost would strengthen the practical case.
Questions for authors
Do paired confidence intervals for downstream F1 differences support a reliable improvement over Reverse and RandomSearch-5, particularly on MuSiQue?
3 comments
Dear Reviewer,
We would like to kindly follow up on our rebuttal and the additional experimental results we provided to address your concerns.
We would greatly appreciate it if you could take another look at our responses and let us know whether any questions or concerns remain.
We are happy to provide further clarification and look forward to your feedback.
Thank you for your time and consideration.
Best regards,
The Authors
Dear Reviewer,
Thank you again for your valuable feedback. As the discussion period nears its end, we would appreciate knowing whether our rebuttal has addressed your concerns or if you have any remaining questions.
Thank you for your time and consideration.
Best regards,
The Authors
We thank the reviewer for recognizing our controlled pair swaps, separation of compression from presentation order, and matched-candidate controls. We address both weaknesses and the downstream QA question with new experiments.
W1. Generalization and whether retained paragraphs preserve the necessary evidence
Response. We expanded MuSiQue to a frozen N=300 cohort, using 10,000 paired question-bootstrap resamples. Against compute-matched RandomSearch-5:
Both nominal intervals are positive: the retention advantage extends beyond the original N=100 MuSiQue evaluation, although it does not resolve the joint scorer–dataset shift.
We tested the concern about paragraph-level metrics directly. On a frozen MuSiQue N=100 subset at 8×, MORSE retains all supporting paragraphs in 42.6% of cases, but only 23.6% pass the stricter decomposition-answer-span chain proxy. The proxy is not semantic proof, and its gains over RandomSearch are not multiplicity-robust. Paragraph recall can overstate complete evidence preservation.
The 100-context extension (N=20) yields MORSE−RandomSearch ΔSF-R of +0.0167 [−0.1250, 0.1417] at 8× and +0.1117 [0.0200, 0.2100] at a fixed 1,024-token budget. We also completed longer-input panels:
These exploratory results broaden coverage, not a claim of uniform QA gains or resolved joint-shift uncertainty.
W2. Practical quality–cost tradeoff and established compression baselines
Response. Candidate-parallel wall time is not total GPU compute. Our original five-GPU MORSE-5 measurement takes 1.977 wall-seconds but 9.887 allocated GPU-seconds/question. We measured the full compression-to-reader path on matched HotpotQA Native N=30 with warm single-A6000 execution:
Against MORSE, paired F1 intervals for RECOMP Extractive and Abstractive both include zero: MORSE−RECOMP is +0.0643 [−0.1190, 0.2500] and +0.0298 [−0.1417, 0.2048], respectively. These systems differ in mechanism and realized budgets: this is a measured system-cost comparison, not an equal-FLOP claim. Historical core timings have isolation telemetry for 22/30 questions. Here, NoCompression is faster and higher-F1 by point estimate than MORSE. MORSE's ordering-mechanism contribution does not require a claim of universal deployment superiority.
Q1. Do paired downstream F1 confidence intervals demonstrate a reliable improvement, especially on MuSiQue?
A1. We paired identical questions, averaged search seeds within each question, and used 10,000 question-level bootstrap replicates rather than treating seeds as independent samples. The expanded MuSiQue N=300 results are:
All F1 and EM intervals include zero. Across 27 exploratory F1 settings, only 5/27 against Reverse and 4/27 against RandomSearch-5 have strictly positive nominal 95% intervals; these are not multiplicity-adjusted discoveries. These results support evidence-retention gains, not a reliable downstream QA improvement. We will preserve that distinction in the revision.
We thank the reviewer for helping us strengthen the statistical analysis and clarify the method's practical limits.