- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that post-retrieval context compressors keep sentences that stay topically relevant after an answer-critical query condition (mostly a date) is changed, even though those sentences no longer support an answer. To measure this, the authors build a fixed-evidence diagnostic of 412 pairs from StreamingQA and TimeQA: one condition of an answerable query is edited so that the same document no longer supports it, evidence sentences are labeled by a blinded LM annotator, and a manual audit confirms the intended support transition for 395 of the 412 pairs. Off-the-shelf OpenProvence, EXIT and Sentinel keep 55% to 72% of the now-invalid evidence.
ConSieve is OpenProvence-Large (a 315M ModernBERT encoder with a token pruning head) fine-tuned on about 22.7k StreamingQA/TimeQA rows. The new ingredient is that LLM-generated counterfactual queries are paired with the unchanged document and every document token in that row gets a PRUNE label. On the diagnostic, ConSieve keeps 12.8% of invalidated evidence. A matched ablation that puts the source rows back in place of 2,605 counterfactual rows gives 56.1%. On a human-written, non-temporal diagnostic built from SQuAD 2.0, ConSieve is on par with OpenProvence overall, better on entity and number edits and worse on antonym and negation edits. In end-to-end RAG (BM25 top-25 over WMT news plus the TimeQA documents, qwen-plus reader, 256-token cap), ConSieve reaches 0.631 token F1 on a 1,500-query StreamingQA/TimeQA set against 0.608 for OpenProvence, at essentially the same compressor latency as OpenProvence.
Critical strengths
-
The fixed-evidence diagnostic is a clean measurement. Holding the document and the candidate sentences fixed and changing only the query means that any change in what a compressor keeps can be attributed to the query, independent of retrieval and of whether the reader could recover the answer anyway. Separating a threshold-free within-pair score (Pair Discrimination) from operating-point retention is the right call, and the paper explains on p. 3 why the two can disagree. The finding that three compressors built on different principles still keep 55% to 72% of evidence that has become invalid (Table 1) is a result that people designing compressors can use directly, and the manual audit of all 412 pairs makes it more credible than a purely LLM-labeled set would be.
-
The experimental protocol is careful, and the paper reports results that cut against the method. All practical runs share the same BM25 top-25 lists, the same reader, the same tokenizer-based budget accounting and the same whole-sentence packing, and mean context lengths at the 256-token cap fall between 222.6 and 226.4 tokens for the sentence-level methods (244.1 for LongLLMLingua), so compression ratio and reader are matched. Thresholds and the budget are chosen on validation data, the MAIN1500 differences come with paired bootstrap intervals, the reader runs are dated, and Table 9 shows ConSieve adds no latency over OpenProvence (3.27 vs 3.25 s median) while EXIT and LongLLMLingua are an order of magnitude slower. At 256 tokens ConSieve (0.631 F1) is also above the uncompressed BM25 top-25 context at about 31k tokens (0.614, Table 8), which OpenProvence at the same cap (0.608) is not. The matched ablation (same initialization, example order, weighting and schedule) is better designed than most I see in this area. I also appreciated that the SQuAD and TimeQA results are reported even though they do not favor ConSieve.
Critical weaknesses
-
The evidence supports sensitivity to the training edit distribution more strongly than general constraint sensitivity. Counterfactuals are predominantly temporal LM edits of StreamingQA/TimeQA; the diagnostic's generation procedure and example pairs are insufficiently specified. Even there, invalid retention is 0.093 on StreamingQA but 0.445 on TimeQA (Table 6). On human-written SQuAD edits, the matched CF versus no-CF improvement is only 0.010 with interval [-0.001, 0.021], while ConSieve versus OpenProvence is worse on negation and antonym with intervals excluding zero. The large in-distribution effect therefore does not clearly transfer. The main diagnostic also makes entire documents unanswerable, where the target is to prune everything, rather than selecting among competing evidence. Appendix E provides useful positive evidence in the latter direction, especially complete switch (+0.063 [0.008, 0.118]), but the correct-transition interval includes zero. These qualifications support a narrower temporal-edit claim, not the general mechanism implied by the title.
-
The practical QA gains are not separated from ordinary in-domain fine-tuning. Public checkpoints are compared against ConSieve trained on the same two datasets used for practical evaluation. The no-CF arm already matches its positive-query F1 (0.743 versus 0.744) and improves invalid retention substantially. Downstream, CF versus no-CF is reported only for GT406 at 256 tokens: 0.622 versus 0.606, single-seed estimates without intervals. The no-CF arm is absent on MAIN1500, so its headline +0.024 F1 cannot be attributed to counterfactual supervision; the Judge difference interval also includes zero. The paper does not quantify how often practical retrieved contexts contain the diagnosed invalid-but-topically-relevant evidence, nor whether training to prune entire documents harms other QA settings. A matched MAIN1500 comparison is central to the claimed practical benefit.
Questions for authors
-
Which model generated the counterfactual queries and which model annotated evidence, for the training data and for the diagnostic? Were the same model and prompt used for both? Could you give the distribution of edit types among the 412 diagnostic pairs and the 2,605 training counterfactuals, and two or three real q⁺/q⁻ pairs? In StreamingQA pairs, does the edit change a date inside the question, and how often does the edited date fall after the "Today is" reference date or outside the date range mentioned in the document? If such surface cues are common, the in-distribution results in Tables 1, 2 and 6 say less about constraint checking than the paper suggests.
-
If available, could you report the no-CF arm on MAIN1500 at the 256-token cap, with paired intervals against the CF arm and against OpenProvence? This is the comparison that decides whether the downstream gain comes from counterfactual supervision or from in-domain fine-tuning, and a clear CF over no-CF gap there would largely resolve my second concern. Separately, please give the exact GT406 values behind Figure 3 for OpenProvence and ConSieve at the 256-token setting, and the family-level CF vs no-CF invalid retention on SQuAD, which only re-slices outputs you already have.
-
Could you split Table 3 by source (1,294 StreamingQA vs 206 TimeQA queries) and say, even approximately, for what fraction of MAIN1500 queries the top-25 contains a sentence that is on-topic but violates the query's temporal condition? Relatedly, under the hard cap ConSieve ranks all sentences and fills the budget (Appendix D.2), so the thresholded invalid retention of Table 1 does not map directly onto the practical setting. Is there any measurement of where invalidated evidence ranks relative to valid evidence from other documents?
Suggestions for future work, not conditions for my assessment: training with human-written or more varied edit families (the SQuAD2-CR families the paper already uses would be a natural start), evaluating on a retrieval setting where valid and invalid evidence for the same question co-occur across documents, and reporting the effect of counterfactual training on standard RAG QA benchmarks.
2 comments
Q2. Non-temporal matched ablation
The held-out human-written SQuAD diagnostic tests non-temporal transfer without SQuAD fine-tuning. Matched CF improves entity valid recall and invalid rejection; numeric edits improve rejection at a recall cost, while negation worsens. The overall invalid-retention reduction is modest, so we characterize transfer as family dependent.
| SQuAD edit family | Valid recall, no-CF → CF | Invalid retention, no-CF → CF |
|---|---|---|
| Overall | 0.942 → 0.954 | 0.635 → 0.625 |
| Entity | 0.948 → 0.960 | 0.541 → 0.514 |
| Antonym | 0.948 → 0.956 | 0.696 → 0.687 |
| Negation | 0.921 → 0.945 | 0.785 → 0.831 |
| Non-temporal number | 0.945 → 0.927 | 0.482 → 0.378 |
Q3. Explaining budget effects and testing ranking directly
Source composition explains the larger GT406 gains: TimeQA accounts for 50.7% of GT406 but 13.7% of MAIN1500. Reweighting MAIN1500 source effects to the GT406 mix yields +0.0776 F1, close to Figure 3's +0.0802. On MAIN1500, the F1 difference is +0.0039 [−0.0104, 0.0183] for 1,294 StreamingQA questions and +0.1490 [0.0922, 0.2054] for 206 TimeQA questions; the corresponding TimeQA EM and Judge gains are +0.1117 [0.0485, 0.1748] and +0.1311 [0.0680, 0.1942].
To test the requested rank-based mechanism, we use 95 diagnostic pairs (57 StreamingQA / 38 TimeQA) with the gold document in the original-query BM25 top-25 pool. Three date-balanced matched model pairs score the edited query against that unchanged pool and pack sentences under the paper's 256-token cap without a threshold. Means are averaged over three seeds; nominal 95% intervals use 10,000 document-cluster bootstrap draws. These models are separate from the Table 3 deployment checkpoint.
Dear Reviewer GKSD,
Thank you for the concrete shortcut, fine-tuning, and hard-budget questions. We address each with targeted controls.
W1/Q1. Testing date-cue shortcuts
We tested two complementary shortcut-resistant slices with frozen matched models. On 39 exploratory StreamingQA pairs excluding the tested future-year and explicit document-range cues (both queries contain years, and the edited year precedes the reference date and lies within the document's stated range), CF reduces invalid retention from 0.637 to 0.115 (−0.5214 [−0.6752, −0.3718]); valid recall is 0.879 vs 0.864 (no-CF vs CF). On a broader 114-pair slice not separated by the tested future-date, document-range, or year-absence rules, CF reduces invalid retention from 0.6704 to 0.2809 (−0.3896 [−0.4830, −0.2944]), with no resolved q+ recall difference (−0.0082 [−0.0487, 0.0294]). A validation-frozen year heuristic also performs much worse than ConSieve on strict pair discrimination among 171 both-year pairs (0.210 vs 0.959). The tested date cues thus do not explain the full rejection effect.
Evidence redirection beyond all-PRUNE negatives
On 255 same-document pairs where both queries are answerable, full-data ConSieve improves complete evidence switching over OpenProvence from 0.349 to 0.412 (+0.0627 [0.0076, 0.1179]). This is a full-model comparison, rather than a CF-specific causal attribution, and demonstrates evidence redirection beyond the all-PRUNE document-unanswerability setup.
Q1. Edit distribution and reproducibility
Among the 2,605 matched-training CF intervention slots (2,316 StreamingQA / 289 TimeQA), direct comparison of the actual query texts identifies 104 year-only substitutions, 2,242 other edits involving year/month tokens, 256 other textual edits, and 3 unchanged queries. These are observed surface-text categories rather than generator-assigned semantic edit labels. Separately, among 371 StreamingQA diagnostic edits, 317 introduce a distinct year (66 year-only); of these, 55 introduce a future year, 188 fall outside the document's explicit year range, and 51 satisfy both cues.
We provide three actual training-set q+/q− pairs below. Each pair shares the same document and reference-date prefix (omitted for readability).
In the Spyker example, the unchanged 2011 document supports the 2010 acquisition of Saab, while 2008 also appears in the document, but only in vehicle-production statistics. The edited year thus precedes the reference date, falls within the document's stated year range, and is explicitly present in the text—yet does not support the requested acquisition. In the other examples, the documents associate the Home Secretary appointment with 1965 and identify Kunming as Yunnan's capital, respectively.
qwen-maxgenerates CF candidates and annotates training evidence;qwen-plusverifies diagnostic evidence and serves as the downstream generator/Judge. The generation template requests one answer-critical condition edit while preserving temporal granularity, and manual auditing confirmed 395/412 intended diagnostic support transitions (Appendix B.1).W2/Q2. Matched-training QA at 256 tokens
Table 3 reports the frozen deployment evaluation, whereas the MAIN1500 rows here report a separate hosted evaluation of the matched-training checkpoints, with OpenProvence re-evaluated as the reference; the absolute scores therefore differ across the two evaluations. On the MAIN1500 common pairs, the matched CF arm improves F1 over no-CF (0.6362 vs 0.6249) and both exceed OpenProvence (0.6190).