acceptodds
Under review as a conference paper at ICLR 2027

MORSE: Multi-Context Ordering via Reverse Scoring for Evidence-Preserving Compression

Abstract

Retrieval-augmented generation often relies on multiple retrieved contexts that contain substantial redundancy, motivating context compression to preserve useful information under limited input budgets. Likelihood-based compressors can account for cross-context redundancy through sequential scoring, but this makes evidence scores dependent on context order. We show that permuting the same contexts under an unchanged compressor can substantially change which supporting evidence survives compression. We attribute this sensitivity to information preemption: earlier, partially relevant contexts can absorb credit for shared information, reducing the incremental scores of later, stronger evidence and increasing its risk of removal. Controlled pair-swap interventions provide direct empirical support for this mechanism by showing that placing stronger evidence before overlapping, partially relevant contexts can improve its survival. Based on this insight, we introduce MORSE, a compression-aware method for evidence-preserving context ordering. MORSE uses reverse query likelihood to construct an evidence-first anchor and to evaluate compressed candidate outputs, enabling compression-aware selection among alternative permutations. Across multi-hop Question Answering (QA) benchmarks, compression procedures, budgets, and scoring models, MORSE improves evidence retention over reverse ordering and generally outperforms matched random search, with downstream QA gains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
context compression, context ordering, information preemption, evidence preservation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eFquOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how context ordering affects evidence retention in sequential likelihood-based compression. It identifies information preemption, where earlier overlapping contexts reduce the scores of later supporting evidence. MORSE combines a reverse-query-likelihood ordering with random permutations and selects among their compressed outputs. Experiments on HotpotQA, 2WikiMultiHopQA, and MuSiQue show improved evidence retention, with less consistent downstream QA gains.

Critical strengths
  1. The controlled pair swaps and separation of compression order from presentation order provide useful evidence that ordering affects which facts survive compression. This makes the analysis informative beyond the proposed method.

  2. The evaluation includes matched-candidate random search, selection-objective ablations, and multiple scoring models and compression procedures. These controls help substantiate the evidence-retention improvements.

Critical weaknesses
  1. Generalization beyond the main settings remains uncertain. Most additional evaluations use 100 examples, and the advantage over RandomSearch-5 is statistically inconclusive under the joint scorer-dataset shift and many context-scaling settings. MuSiQue’s paragraph-level retention metric also does not establish that the necessary facts survive.

  2. The practical quality-cost tradeoff is not fully established. Table 2 reports 1.89 seconds for MORSE-5 versus 0.21 seconds for Reverse, despite candidate parallelism across five GPUs. Given the mixed downstream QA improvements, comparisons with established compression systems at matched total cost would strengthen the practical case.

Questions for authors

Do paired confidence intervals for downstream F1 differences support a reliable improvement over Reverse and RandomSearch-5, particularly on MuSiQue?

3 comments

We thank the reviewer for recognizing our controlled pair swaps, separation of compression from presentation order, and matched-candidate controls. We address both weaknesses and the downstream QA question with new experiments.

W1. Generalization and whether retained paragraphs preserve the necessary evidence

Response. We expanded MuSiQue to a frozen N=300 cohort, using 10,000 paired question-bootstrap resamples. Against compute-matched RandomSearch-5:

MuSiQueNMORSE−RandomSearch ΔSP-R [95% CI]
8×300+0.0115 [0.0006, 0.0224]
16×300+0.0227 [0.0092, 0.0365]

Both nominal intervals are positive: the retention advantage extends beyond the original N=100 MuSiQue evaluation, although it does not resolve the joint scorer–dataset shift.

We tested the concern about paragraph-level metrics directly. On a frozen MuSiQue N=100 subset at 8×, MORSE retains all supporting paragraphs in 42.6% of cases, but only 23.6% pass the stricter decomposition-answer-span chain proxy. The proxy is not semantic proof, and its gains over RandomSearch are not multiplicity-robust. Paragraph recall can overstate complete evidence preservation.

The 100-context extension (N=20) yields MORSE−RandomSearch ΔSF-R of +0.0167 [−0.1250, 0.1417] at 8× and +0.1117 [0.0200, 0.2100] at a fixed 1,024-token budget. We also completed longer-input panels:

N=20 panelReverse F1RandomSearch F1MORSE F1
MultiHop-RAG real retrieval0.8500.8000.800
MultiHop-RAG evidence-present0.8000.8500.800
FRAMES gold-document0.3550.2780.305

These exploratory results broaden coverage, not a claim of uniform QA gains or resolved joint-shift uncertainty.

W2. Practical quality–cost tradeoff and established compression baselines

Response. Candidate-parallel wall time is not total GPU compute. Our original five-GPU MORSE-5 measurement takes 1.977 wall-seconds but 9.887 allocated GPU-seconds/question. We measured the full compression-to-reader path on matched HotpotQA Native N=30 with warm single-A6000 execution:

MethodF1Retained tokensE2E s/question
NoCompression0.62461171.40.329
Reverse-10.5469142.82.044
RandomSearch-50.6100142.68.306
MORSE-50.6052142.18.466
RECOMP Extractive0.5410142.00.179
RECOMP Abstractive0.575555.01.364
LongLLMLingua0.3319184.50.408

Against MORSE, paired F1 intervals for RECOMP Extractive and Abstractive both include zero: MORSE−RECOMP is +0.0643 [−0.1190, 0.2500] and +0.0298 [−0.1417, 0.2048], respectively. These systems differ in mechanism and realized budgets: this is a measured system-cost comparison, not an equal-FLOP claim. Historical core timings have isolation telemetry for 22/30 questions. Here, NoCompression is faster and higher-F1 by point estimate than MORSE. MORSE's ordering-mechanism contribution does not require a claim of universal deployment superiority.

Q1. Do paired downstream F1 confidence intervals demonstrate a reliable improvement, especially on MuSiQue?

A1. We paired identical questions, averaged search seeds within each question, and used 10,000 question-level bootstrap replicates rather than treating seeds as independent samples. The expanded MuSiQue N=300 results are:

RatioMORSE vsΔF1 [95% CI]ΔEM [95% CI]
8×Reverse+0.0124 [−0.0169, 0.0411]+0.0067 [−0.0240, 0.0340]
8×RandomSearch-5+0.0076 [−0.0091, 0.0251]+0.0060 [−0.0107, 0.0233]
16×Reverse−0.0049 [−0.0308, 0.0216]−0.0100 [−0.0333, 0.0127]
16×RandomSearch-5+0.0053 [−0.0117, 0.0224]+0.0013 [−0.0140, 0.0167]

All F1 and EM intervals include zero. Across 27 exploratory F1 settings, only 5/27 against Reverse and 4/27 against RandomSearch-5 have strictly positive nominal 95% intervals; these are not multiplicity-adjusted discoveries. These results support evidence-retention gains, not a reliable downstream QA improvement. We will preserve that distinction in the revision.

We thank the reviewer for helping us strengthen the statistical analysis and clarify the method's practical limits.

Dear Reviewer,

We would like to kindly follow up on our rebuttal and the additional experimental results we provided to address your concerns.

We would greatly appreciate it if you could take another look at our responses and let us know whether any questions or concerns remain.

We are happy to provide further clarification and look forward to your feedback.

Thank you for your time and consideration.

Best regards,

The Authors

Dear Reviewer,

Thank you again for your valuable feedback. As the discussion period nears its end, we would appreciate knowing whether our rebuttal has addressed your concerns or if you have any remaining questions.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_Pk5FOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors explore how document ordering affects which evidence survives context compression in multi-hop QA. Specifically, they observe a phenomenon called information preemption, where earlier, partially relevant documents can reduce the importance scores of later, more informative evidence. To address this, they propose MORSE, which uses reverse query likelihood to order documents and selects the best compressed output among multiple candidate orderings. They conduct the experiments mainly on two multi-hop QA benchmarks.

Critical strengths
  • The observation that document ordering can affect which evidence survives compression is interesting, although the reverse query likelihood-based ordering has already been explored in prior work (e.g., LongLLMLingua).

  • Some experimental analyses such as different compression ratios and pair-swap experiments are detailed, though they are still limited to relatively simple QA datasets.

Critical weaknesses
  • The novelty is unclear, specifically when compared to LongLLMLingua, which already also uses reverse query likelihood during compression. As the motivation (compression) and main strategy (reverse query likelihood) are highly similar to the previous work, what is the main methodological contribution is, beyond the (relatively straightforward) engineering effort such as evaluating multiple candidate orderings and selecting the best compressed output?

  • The evaluation scope seems to be limited, specifically the experiments are conducted on (relatively) old and conventional multi-hop QA benchmarks (HotpotQA and 2WikiMultiHopQA). I would particularly like to highlight the blog post regarding it [1]. While there is an additional evaluation on MuSiQue, this is only included in a smaller-scale generalization experiment, not as the main experiments. I believe that more recent and challenging multi-hop reasoning benchmarks would be necessary (as the main experiments).

  • Recent long-context LLMs are capable of digesting much longer inputs (e.g., hundreds of thousands or even millions of tokens), whereas most experiments involve only 10-20 relatively short documents (each truncated to at most 180 tokens), which questions the practical significance of the proposed approach. There is actually previous work on investigating the capability of long context LLMs on retrieval settings [2], and even on the compression on such settings [3]. Therefore, it is highly unclear whether context compression is even necessary in such small-scale settings, when considering much increased computational efforts of the proposed approach.

  • Related to the previous point, I’m also concerned about the efficiency, as MORSE needs much more additional computational overhead to a simpler baselines, such as Reverse ordering.

  • Wouldn’t simply reranking documents or sentences using stronger reranking models could achieve similar improvements, without the additional computational overhead of evaluating multiple candidate orderings?

  • Also, the baselines on RAG compression such as EXIT and RECOMP are not experimentally compared against.

  • What is the discrepancy between improvements in SF-R and downstream QA performance? While MORSE consistently improves SF-R, the corresponding improvements in QA performance are (relatively) small, which raises the question of whether preserving more annotated supporting facts actually translates into better answer generation, which is the ultimate goal or objective of the QA task.

  • While the paper shows that document order can affect which evidence is preserved after compression, the explanation of information preemption is not very clear. For example, an earlier document may contain some relevant information (but not the actual answer), while a later document contains additional important information (including the answer). Even if the two documents share some information, it is unclear why the later document would receive a substantially lower importance score (and potentially be removed), despite containing new and useful information.

[1] Why You Should Stop Using HotpotQA for AI Agents Evaluation in 2025.

[2] Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?

[3] Efficient Long Context Language Model Retrieval with Compression

Questions for authors

See above.

4 comments

Part 1/2 — Novelty, evaluation scope, and efficiency (W1–W4)

We thank the reviewer for recognizing that context ordering changes which evidence survives compression. We agree that reverse query likelihood and document reordering have appeared in LongLLMLingua; our claim is not that these primitives are new. The central contribution is identifying information preemption inside sequential compression and showing how to optimize against it. We address the eight concerns separately.

W1. What is new beyond LongLLMLingua?

Response. Characterizing MORSE as "reverse likelihood + random search" describes its ingredients but misses the problem it identifies. LongLLMLingua uses query-aware scoring and reordering; it does not establish our central mechanism: given identical documents and budgets, earlier overlapping context can suppress the conditional importance assigned to later, uniquely useful evidence. Controlled pair swaps and canonicalized reader presentation isolate the effect during compression, rather than conflating it with downstream position bias.

MORSE operationalizes this finding by evaluating actual compressed outputs across candidate orders and selecting with JBJ_B, instead of assuming that a static relevance ranking is sufficient. Our additional natural-overlap control (N=50, 8×) shows a high-minus-low overlap difference of +0.1410 [0.0547, 0.2307] in evidence-first SF-R rescue gain. This supports a new mechanism and compression-aware optimization problem, not novelty in random sampling itself.

W2. Conventional benchmarks and limited MuSiQue coverage

Response. We expanded MuSiQue to N=300 and added two more challenging long-input settings, each with a complete frozen N=20 cohort:

EvaluationNNoCompression F1Reverse F1MORSE F1
MultiHop-RAG, real retrieval200.46650.85000.8000
MultiHop-RAG, gold-evidence-present200.52860.80000.8000
FRAMES, historical gold documents200.40950.35500.3050

The real-retrieval and gold-injected panels are explicitly separated; FRAMES uses screened historical source documents, not real retrieval. These are exploratory evaluations, not full benchmark splits. Crucially, they do not support a claim that MORSE consistently improves F1 over Reverse; they directly address the lack of evaluation beyond the original QA datasets.

W3. Are such short contexts relevant when LLM windows are much larger?

Response. Capacity to accept a context is not evidence that retaining every token is optimal. We therefore tested longer inputs rather than extrapolating from 10–20 short documents. MultiHop-RAG uses approximately 29.5K input tokens/question, compressed to 3.7K; FRAMES uses 18.2K, compressed to 2.3K. On MultiHop-RAG real retrieval, both Reverse (F1 0.8500) and MORSE (0.8000) exceed NoCompression (0.4665) as point estimates. Conversely, on FRAMES, NoCompression (0.4095) exceeds MORSE (0.3050). Thus, long-context support does not eliminate the question of compression, but compression benefit is workload-dependent—not an automatic justification for MORSE.

W4. Does the added compute justify the benefits?

Response. We measured the entire compression–reader path on a matched Hotpot Native N=30 cohort, using a warm single RTX A6000:

MethodF1E2E seconds/question
NoCompression0.62460.3293
Reverse-10.54692.0442
Qwen3B Reverse-10.51365.1708
MORSE-50.60528.4659

A stronger scorer does not improve the observed quality–cost balance here; MORSE is not preferable to NoCompression on this short-input workload. These are real measurements, not FLOP-matched claims; 22/30 historical core timing records have continuous isolation telemetry. Our contribution is the measured ordering failure mode and a way to mitigate it, not a claim of universal end-to-end speedup. We will make this scope explicit in the limitations.

Part 2/2 — Reranking, compressor baselines, QA, and mechanism (W5–W8)

W5. Could stronger reranking replace MORSE?

Response. We tested BGE-reranker-v2-m3, both as a single-pass ordering and as the anchor for a five-candidate search. On Hotpot Native, N=500, 8×:

ComparisonΔSF-R [95% CI]
BGE Rank-1 − MORSE-5−0.0312 [−0.0534, −0.0087]
BGE Anchor-5 − MORSE-5+0.0014 [−0.0107, 0.0135]

BGE Rank-1's ΔF1 is −0.0125 [−0.0423, 0.0172]. Single-pass learned ranking does not recover MORSE's evidence retention in this setting, whereas replacing Reverse with a BGE anchor inside the same five-candidate search is competitive. This supports our distinction between relevance-based ordering and evaluating what the compressor actually retains. It does not establish a universal downstream F1 advantage.

W6. Missing EXIT and RECOMP compression baselines

Response. We implemented the official RECOMP Extractive and Abstractive systems and LongLLMLingua. On the identical Hotpot Native N=30, nominal 8× cohort:

MethodF1SF-RRetained tokensE2E s/question
MORSE-50.60520.5567142.18.466
RECOMP Extractive0.54100.6011142.00.179
RECOMP Abstractive0.5755N/A55.01.364
LongLLMLingua0.3319N/A184.50.408

The paired MORSE-minus-RECOMP F1 differences are +0.0643 [−0.1190, 0.2500] (extractive) and +0.0298 [−0.1417, 0.2048] (abstractive): neither establishes QA superiority. RECOMP Extractive has higher SF-R and much lower cost here; it is a different learned sentence-selection compressor, not an ordering-optimization baseline. LongLLMLingua's F1 contrast favors MORSE (+0.2733 [0.0856, 0.4556]), but realized output lengths differ. Exact sentence-level SF-R is not valid for partial-token or abstractive outputs. EXIT could not be evaluated because its official required checkpoint is access-gated; we will document this rather than report an inauthentic substitute.

W7. Why does higher SF-R not translate to higher QA?

Response. We directly tested this rather than assuming retention predicts answer quality. On MuSiQue N=300, against matched RandomSearch-5:

RatioΔSP-R [95% CI]ΔF1 [95% CI]
8×+0.0115 [0.0006, 0.0224]+0.0076 [−0.0091, 0.0251]
16×+0.0227 [0.0092, 0.0365]+0.0053 [−0.0117, 0.0224]

On a fixed N=100 subset at 8×, 42.6% of MORSE outputs retain all gold supporting paragraphs, but only 23.6% preserve all official decomposition-answer strings in a stricter chain proxy. Paragraph recall therefore overstates complete-chain coverage; the proxy itself is not a semantic proof, and its gains are not multiplicity-robust. This helps explain why additional retained evidence need not produce a reliable F1 improvement. We will report retention and downstream QA as distinct outcomes.

W8. Why can earlier partial evidence suppress later unique evidence?

Response. A later supporting sentence may contain both shared information and a unique necessary relation. In sequential conditional scoring, earlier overlapping text lowers the incremental importance of the shared component. When importance is aggregated at the sentence level, the total score can fall below competing units under a fixed budget even if the unique relation remains useful. We do not claim that unique information loses all value.

Our additional interventions are consistent with this mechanism:

InterventionEffect [95% CI]
Natural overlap control, 8×, N=50: high−low SF-R rescue gain+0.1410 [0.0547, 0.2307]
Natural overlap control, 16×: high−low SF-R rescue gain+0.0573 [−0.0060, 0.1233]
3B high−low conditional-score recovery, N=40+2.4062 [0.1511, 4.5017]

The 3B score effect remains positive, while removal is not guaranteed across budgets. This separates ordering-induced score changes from the stronger, unsupported claim that all overlapping information necessarily removes gold evidence.

We appreciate the reviewer’s suggestions and will sharpen the novelty statement and scope of our conclusions accordingly.

Dear Reviewer,

We would like to kindly follow up on our rebuttal and the additional experimental results we provided to address your concerns.

We would greatly appreciate it if you could take another look at our responses and let us know whether any questions or concerns remain.

We are happy to provide further clarification and look forward to your feedback.

Thank you for your time and consideration.

Best regards,

The Authors

Dear Reviewer,

Thank you again for your valuable feedback. As the discussion period nears its end, we would appreciate knowing whether our rebuttal has addressed your concerns or if you have any remaining questions.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_CjZpOpenReview ↗
Rating
2
Confidence
2
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies context ordering as a decision variable in sequential likelihood-based (query-conditioned) compression for RAG. The authors show that permuting the same retrieved contexts substantially changes which evidence survives compression (supporting-fact recall varies widely across 200 random permutations on HotpotQA), attribute this to "information preemption" — earlier, partially relevant contexts absorbing credit for shared information and suppressing later, stronger evidence — and support this mechanism with controlled pair-swap interventions. They propose MORSE: a reverse-query-likelihood evidence-first anchor, plus running the actual compressor on K candidate permutations and selecting the output that maximizes the reverse query evidence of the retained content. Across HotpotQA, 2WikiMultiHopQA, MuSiQue, the TRUE-20 stress test, two compressors (1P, ISD), and two scorer models, MORSE improves evidence retention over static reverse ordering and compute-matched random search, with downstream QA gains that are positive but less uniform.

Critical strengths
  • Identifies a real, previously overlooked phenomenon — ordering effects inside the compression process itself (distinct from presentation-order effects) — and provides a convincing mechanistic account (information preemption) validated by controlled pair-swap interventions, including a quantitative suppression analysis.
  • Methodologically careful evaluation: compute-matched baselines (MORSE-5 vs RANDOMSEARCH-5), clean separation of compression-stage vs presentation-stage effects (Sec. 6.6), ablations of search budget and selection objective, and cross-scorer/cross-dataset generalization checks with confidence intervals.
  • Honest reporting: the paper does not oversell downstream gains, explicitly noting that higher SF-R does not always translate into higher F1/EM, and reports where MORSE's advantage over random search is statistically inconclusive.
Critical weaknesses
  • The gains are modest and the method is simple: SF-R improvements are typically a few points over strong static baselines, downstream QA gains are inconsistent (and sometimes negative under ISD), and the method is essentially "sort by reverse likelihood + random permutation search with a selection objective," which limits technical novelty and practical impact; there is no dedicated limitations section.
  • The random search component appears to do much of the work: MORSE-5's advantage over the compute-matched RANDOMSEARCH-5 is small in many settings and statistically inconclusive under the joint scorer–dataset shift, raising questions about what the anchor actually adds.
  • The mechanism is demonstrated only with small likelihood scorers (0.5B–1B); it is unclear whether information preemption persists with LLM-based or learned compressors (e.g., LLMLingua-2-style), which are the practically dominant family. The theoretical decomposition (Eq. 4–5) is illustrative rather than predictive.
  • The 5× compression-cost overhead (1.89s vs 0.21s per example) is not weighed against alternatives, e.g., simply using a stronger scorer or spending the same compute on more compression iterations; evaluation scale is also limited (500/100 examples).
Questions for authors
  • Can you characterize when higher SF-R fails to translate into downstream QA gains (e.g., milder compression, ISD), and give practitioners guidance on when MORSE's extra cost is justified?
  • How much of the gain comes from the reverse anchor versus the random search? Would a smarter candidate-generation strategy (greedy swaps, diversification-aware sampling) substantially beat random candidates?
  • Does information preemption occur with LLM-based/learned compressors and larger scoring models? Your scorers are 0.5B–1B; do the ordering effects transfer?
  • Have you tested MORSE on larger context collections (50–100+ contexts), and how accurately does the reverse-evidence selection objective J_B track SF-R across budgets in those regimes?
5 comments

Part 1/2 — Critical Weaknesses (W1–W4)

We thank the reviewer for recognizing a "real, previously overlooked phenomenon" and the strength of our controlled interventions. Our central point is that MORSE is motivated by a newly identified failure mode inside sequential compression, not by a claim that reverse ranking or random sampling is new. We address each concern and question separately.

W1. Novelty, effect size, and limitations

Response. We respectfully disagree that MORSE's contribution is adequately captured by "reverse ranking + random search." That describes the implementation, but not the problem it uncovers. With the same documents and token budget, changing compression order changes the conditional credit assigned to supporting evidence and can change what survives. Controlled pair swaps and canonical downstream presentation isolate this compression-stage effect from reader position bias. In a new overlap control (N=50, 8×), high versus low overlap changes the evidence-first SF-R rescue gain by +0.1410 [0.0547, 0.2307]. This is evidence for information preemption, not merely another relevance-ordering heuristic.

MORSE operationalizes the finding by evaluating candidate orders through the actual compressor and selecting by the retained-output objective JBJ_B. We do not claim the search heuristic is optimal. We will add a dedicated limitations section distinguishing this mechanistic contribution from downstream QA and deployment claims.

W2. Does random search account for the gains?

Response. We isolated the Reverse anchor's contribution under a matched five-candidate budget: it is selected in 24–49% of cases and improves mean retention in several settings. Importantly, on expanded MuSiQue N=300, MORSE exceeds matched RandomSearch-5 in supporting-paragraph recall at 8×: +0.0115 [0.0006, 0.0224] and 16×: +0.0227 [0.0092, 0.0365]. Thus, the informed anchor is not interchangeable with an additional random proposal in every setting. We also test smarter proposals directly in Q2; those results do not assume the original MORSE proposal strategy is optimal.

W3. Scorer/compressor scope and explanatory theory

Response. We directly tested the proposed overlap-specific suppression, rather than inferring mechanism from MORSE's aggregate performance. It persists with a 3B scorer on 40 matched questions (Q3). This is a score-level result; the effect on actual evidence removal depends on the compression budget. Conversely, tested LLMLingua-2 and RECOMP Extractive configurations are order-invariant, so we do not claim universality across learned compressors. Eqs. (4–5) explain why conditional importance can fall; they are not presented as a theorem predicting every removal decision.

W4. Cost versus alternatives and evaluation scale

Response. We measured the full compressor-to-reader path on matched Hotpot Native N=30 with a warm single A6000:

MethodF1E2E s/question
NoCompression0.62460.3293
Reverse-10.54692.0442
3B Reverse-10.51365.1708
RandomSearch-50.61008.3056
MORSE-50.60528.4659

On this short-input workload, NoCompression has the better F1/cost point estimate. This is an applicability limit, not a reason to dismiss the measured compression-stage phenomenon. Increasing scorer size alone does not yield a better observed F1/cost tradeoff here. Costs are measured, not inferred from model size or candidate count; core timing isolation telemetry covers 22/30 historical records. We also extended evaluation to MuSiQue N=300, 100 contexts, MultiHop-RAG, and FRAMES. Q4 reports what the scaling evidence does—and does not—establish.

Part 2/2 — Questions for Authors (Q1–Q4)

Q1. When does higher SF-R fail to improve QA, and when is MORSE worth its cost?

A1. Paired question-bootstrap analysis across 27 exploratory settings finds strictly positive nominal F1 intervals in only 5/27 comparisons against Reverse and 4/27 against RandomSearch-5. The expanded MuSiQue evaluation (N=300) makes the distinction concrete:

RatioMORSE−RandomSearch ΔSP-R [95% CI]ΔF1 [95% CI]
8×+0.0115 [0.0006, 0.0224]+0.0076 [−0.0091, 0.0251]
16×+0.0227 [0.0092, 0.0365]+0.0053 [−0.0117, 0.0224]

We analyzed the 1P/ISD and 4×/8×/16× regimes separately. On a fixed MuSiQue N=100 subset at 8×, 42.6% retain every supporting paragraph, but only 23.6% pass a stricter decomposition-answer-span chain proxy. This proxy is not a semantic proof; it shows why improved paragraph retention need not preserve a usable multi-hop chain. MORSE is justified only where measured evidence/token-budget requirements outweigh compression costs; we do not claim an automatic QA benefit.

Q2. How much does Reverse add, and can greedy/diversified proposals do better?

A2. Reverse is selected in 24–49% of cases and improves retention in some matched-search settings (W2). We implemented LocalSwap, diversified proposals, and GreedySwap-5 using the same scorer, compressor, JBJ_B, and at most five evaluated candidates, without gold labels or hidden model-scored proposals:

Hotpot Native, 8×NGreedySwap−MORSE ΔSF-R [95% CI]
Ordering study50+0.0780 [0.0027, 0.1563]
Matched system subset30+0.1106 [0.0139, 0.2133]

The N=30 sample is nested in N=50, and its improvement is not robust to the declared three-metric correction. Its F1 change is −0.0583 [−0.1667, 0.0417]. Thus, smarter search can improve evidence retention, but neither the original proposals nor their replacement establishes reliable QA gains.

Q3. Does information preemption transfer to larger scorers or learned compressors?

A3. In 40 matched HotpotQA TRUE-20 questions, the high-minus-low-overlap conditional-score recovery is:

ScorerEffect [95% CI]
0.5B+6.8962 [5.0186, 8.8568]
3B+2.4062 [0.1511, 4.5017]

The 3B effect remains positive, though smaller on the raw score scale; attenuation is not invariant to all normalizations. At 3B/8×, the overlap-specific evidence-rescue contrast is +20 percentage points [5, 35]; 16× rescue and SF-R contrasts are inconclusive. LLMLingua-2 and tested RECOMP Extractive are order-invariant; LongLLMLingua changes native outputs in 16/20 orderings, but this alone does not establish the same mechanism. Our claim is therefore specific to the tested sequential likelihood family.

Q4. What happens at 100 contexts, and how reliable is JBJ_B?

A4. We tested 100 contexts on N=20 fixed questions, with controlled distractor augmentation:

BudgetMORSE−RandomSearch ΔSF-R [95% CI]
8×+0.0167 [−0.1250, 0.1417]
16×+0.0917 [0.0000, 0.1917]
Fixed 1,024 tokens+0.1117 [0.0200, 0.2100]

Only the fixed-budget interval is strictly positive at the nominal level. Mean within-question Spearman ρ between JBJ_B and SF-R is 0.491 (19 valid questions, 8×) and 0.675 (17 valid, fixed budget); candidate-oracle agreement is 80% and mean oracle regret 0.128/0.100, respectively. Thus, JBJ_B is informative but imperfect. These are controlled 100-context tests, not a claim of general superiority in real 100-document retrieval.

We thank the reviewer for helping us clarify the key distinction: a reproducible compression-stage failure mode and an actionable ordering variable, without claiming universal QA or cost superiority.

Dear Reviewer,

We would like to kindly follow up on our rebuttal and the additional experimental results we provided to address your concerns.

We would greatly appreciate it if you could take another look at our responses and let us know whether any questions or concerns remain.

We are happy to provide further clarification and look forward to your feedback.

Thank you for your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Additional Evidence for the Area Chair

Dear Area Chair,

Thank you for overseeing our submission. We would like to briefly summarize the reviewers' main concerns and our responses to facilitate your assessment.

1. Reviewer CjZp — Novelty, Mechanism, and Practical Impact

  • Concern: Modest improvements, unclear contribution beyond random search, limited generalization, and substantial computational overhead.
  • Response: We strengthened the evidence for information preemption through controlled overlap experiments and confirmed the mechanism with a larger 3B scorer. On expanded MuSiQue (N=300), MORSE significantly improves evidence retention over compute-matched RandomSearch-5 at both 8× and 16× compression. We also evaluated smarter candidate generation, 100-context settings, and end-to-end costs, while acknowledging that downstream QA and deployment benefits remain inconsistent.

2. Reviewer Pk5F — Novelty, Benchmark Coverage, and Baselines

  • Concern: Unclear distinction from LongLLMLingua, limited long-context evaluation, missing compression/reranking baselines, and weak correspondence between evidence retention and QA.
  • Response: We clarified that our novelty lies in identifying information preemption within sequential compression, rather than introducing reverse likelihood itself. We added longer-input evaluations on MultiHop-RAG and FRAMES, comparisons with BGE reranking, RECOMP, and LongLLMLingua, and additional pair-swap and overlap analyses. These experiments strengthen the mechanistic evidence while revealing workload-dependent QA outcomes.

3. Reviewer eFqu — Statistical Reliability and Quality–Cost Trade-off

  • Concern: Uncertain generalization, insufficient statistical evidence for downstream QA improvements, and high computational cost.
  • Response: We expanded MuSiQue to 300 examples, obtaining positive paired confidence intervals for supporting-paragraph recall improvements over RandomSearch-5. However, downstream F1 confidence intervals include zero. We further analyzed complete evidence-chain preservation and measured full compression-to-reader latency against established baselines, explicitly acknowledging the substantial overhead and the absence of universal QA superiority.

Overall, the rebuttal substantially strengthens the evidence for our central contribution: context ordering induces information preemption during sequential compression, and compression-aware ordering can mitigate evidence loss. We have also expanded the evaluation, introduced stronger baselines, and clarified the distinction between reliable evidence-retention improvements and less consistent downstream QA benefits.

We sincerely appreciate your consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.