acceptodds
Under review as a conference paper at ICLR 2027

ConSieve: Constraint-Sensitive Evidence Selection for RAG Context Compression

Abstract

Post-retrieval context compressors for retrieval-augmented generation (RAG) typically select evidence based on its semantic relevance to the query. Relevance alone does not guarantee validity because text can remain topically relevant while violating an answer-critical condition, especially under temporal constraints, and therefore fail to support the answer. To measure this failure directly, we construct fixed-evidence counterfactual diagnostic pairs by changing one answer-critical query condition while holding the document and candidate evidence fixed, so evidence that supports the original query becomes invalidated for the edited query. On this diagnostic, the tested baselines retain 55.3% to 72.1% of the invalidated evidence. We introduce ConSieve, a constraint-sensitive extractive compressor trained with counterfactual supervision. On the predominantly temporal diagnostic, ConSieve retains more supporting evidence under the original query than every tested baseline and prunes 87.2% of the same evidence after the edit makes it invalid. A matched ablation links this behavior to counterfactual supervision, reducing invalid retention from 56.1% to 11.4%. Under a shared 256-token context cap in end-to-end RAG, ConSieve reaches 0.631 downstream QA F1 compared with 0.608 for the strongest tested baseline. A separate multi-budget analysis shows the same overall advantage across the tested context budgets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
retrieval-augmented generation, context compression, evidence selection, counterfactual learning, question answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GKSDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that post-retrieval context compressors keep sentences that stay topically relevant after an answer-critical query condition (mostly a date) is changed, even though those sentences no longer support an answer. To measure this, the authors build a fixed-evidence diagnostic of 412 pairs from StreamingQA and TimeQA: one condition of an answerable query is edited so that the same document no longer supports it, evidence sentences are labeled by a blinded LM annotator, and a manual audit confirms the intended support transition for 395 of the 412 pairs. Off-the-shelf OpenProvence, EXIT and Sentinel keep 55% to 72% of the now-invalid evidence.

ConSieve is OpenProvence-Large (a 315M ModernBERT encoder with a token pruning head) fine-tuned on about 22.7k StreamingQA/TimeQA rows. The new ingredient is that LLM-generated counterfactual queries are paired with the unchanged document and every document token in that row gets a PRUNE label. On the diagnostic, ConSieve keeps 12.8% of invalidated evidence. A matched ablation that puts the source rows back in place of 2,605 counterfactual rows gives 56.1%. On a human-written, non-temporal diagnostic built from SQuAD 2.0, ConSieve is on par with OpenProvence overall, better on entity and number edits and worse on antonym and negation edits. In end-to-end RAG (BM25 top-25 over WMT news plus the TimeQA documents, qwen-plus reader, 256-token cap), ConSieve reaches 0.631 token F1 on a 1,500-query StreamingQA/TimeQA set against 0.608 for OpenProvence, at essentially the same compressor latency as OpenProvence.

Critical strengths
  1. The fixed-evidence diagnostic is a clean measurement. Holding the document and the candidate sentences fixed and changing only the query means that any change in what a compressor keeps can be attributed to the query, independent of retrieval and of whether the reader could recover the answer anyway. Separating a threshold-free within-pair score (Pair Discrimination) from operating-point retention is the right call, and the paper explains on p. 3 why the two can disagree. The finding that three compressors built on different principles still keep 55% to 72% of evidence that has become invalid (Table 1) is a result that people designing compressors can use directly, and the manual audit of all 412 pairs makes it more credible than a purely LLM-labeled set would be.

  2. The experimental protocol is careful, and the paper reports results that cut against the method. All practical runs share the same BM25 top-25 lists, the same reader, the same tokenizer-based budget accounting and the same whole-sentence packing, and mean context lengths at the 256-token cap fall between 222.6 and 226.4 tokens for the sentence-level methods (244.1 for LongLLMLingua), so compression ratio and reader are matched. Thresholds and the budget are chosen on validation data, the MAIN1500 differences come with paired bootstrap intervals, the reader runs are dated, and Table 9 shows ConSieve adds no latency over OpenProvence (3.27 vs 3.25 s median) while EXIT and LongLLMLingua are an order of magnitude slower. At 256 tokens ConSieve (0.631 F1) is also above the uncompressed BM25 top-25 context at about 31k tokens (0.614, Table 8), which OpenProvence at the same cap (0.608) is not. The matched ablation (same initialization, example order, weighting and schedule) is better designed than most I see in this area. I also appreciated that the SQuAD and TimeQA results are reported even though they do not favor ConSieve.

Critical weaknesses
  1. The evidence supports sensitivity to the training edit distribution more strongly than general constraint sensitivity. Counterfactuals are predominantly temporal LM edits of StreamingQA/TimeQA; the diagnostic's generation procedure and example pairs are insufficiently specified. Even there, invalid retention is 0.093 on StreamingQA but 0.445 on TimeQA (Table 6). On human-written SQuAD edits, the matched CF versus no-CF improvement is only 0.010 with interval [-0.001, 0.021], while ConSieve versus OpenProvence is worse on negation and antonym with intervals excluding zero. The large in-distribution effect therefore does not clearly transfer. The main diagnostic also makes entire documents unanswerable, where the target is to prune everything, rather than selecting among competing evidence. Appendix E provides useful positive evidence in the latter direction, especially complete switch (+0.063 [0.008, 0.118]), but the correct-transition interval includes zero. These qualifications support a narrower temporal-edit claim, not the general mechanism implied by the title.

  2. The practical QA gains are not separated from ordinary in-domain fine-tuning. Public checkpoints are compared against ConSieve trained on the same two datasets used for practical evaluation. The no-CF arm already matches its positive-query F1 (0.743 versus 0.744) and improves invalid retention substantially. Downstream, CF versus no-CF is reported only for GT406 at 256 tokens: 0.622 versus 0.606, single-seed estimates without intervals. The no-CF arm is absent on MAIN1500, so its headline +0.024 F1 cannot be attributed to counterfactual supervision; the Judge difference interval also includes zero. The paper does not quantify how often practical retrieved contexts contain the diagnosed invalid-but-topically-relevant evidence, nor whether training to prune entire documents harms other QA settings. A matched MAIN1500 comparison is central to the claimed practical benefit.

Questions for authors
  1. Which model generated the counterfactual queries and which model annotated evidence, for the training data and for the diagnostic? Were the same model and prompt used for both? Could you give the distribution of edit types among the 412 diagnostic pairs and the 2,605 training counterfactuals, and two or three real q⁺/q⁻ pairs? In StreamingQA pairs, does the edit change a date inside the question, and how often does the edited date fall after the "Today is" reference date or outside the date range mentioned in the document? If such surface cues are common, the in-distribution results in Tables 1, 2 and 6 say less about constraint checking than the paper suggests.

  2. If available, could you report the no-CF arm on MAIN1500 at the 256-token cap, with paired intervals against the CF arm and against OpenProvence? This is the comparison that decides whether the downstream gain comes from counterfactual supervision or from in-domain fine-tuning, and a clear CF over no-CF gap there would largely resolve my second concern. Separately, please give the exact GT406 values behind Figure 3 for OpenProvence and ConSieve at the 256-token setting, and the family-level CF vs no-CF invalid retention on SQuAD, which only re-slices outputs you already have.

  3. Could you split Table 3 by source (1,294 StreamingQA vs 206 TimeQA queries) and say, even approximately, for what fraction of MAIN1500 queries the top-25 contains a sentence that is on-topic but violates the query's temporal condition? Relatedly, under the hard cap ConSieve ranks all sentences and fills the budget (Appendix D.2), so the thresholded invalid retention of Table 1 does not map directly onto the practical setting. Is there any measurement of where invalidated evidence ranks relative to valid evidence from other documents?

Suggestions for future work, not conditions for my assessment: training with human-written or more varied edit families (the SQuAD2-CR families the paper already uses would be a natural start), evaluating on a retrieval setting where valid and invalid evidence for the same question co-occur across documents, and reporting the effect of counterfactual training on standard RAG QA benchmarks.

2 comments

Dear Reviewer GKSD,

Thank you for the concrete shortcut, fine-tuning, and hard-budget questions. We address each with targeted controls.

W1/Q1. Testing date-cue shortcuts

Q1: If such surface cues are common, the in-distribution results in Tables 1, 2 and 6 say less about constraint checking than the paper suggests.

We tested two complementary shortcut-resistant slices with frozen matched models. On 39 exploratory StreamingQA pairs excluding the tested future-year and explicit document-range cues (both queries contain years, and the edited year precedes the reference date and lies within the document's stated range), CF reduces invalid retention from 0.637 to 0.115 (−0.5214 [−0.6752, −0.3718]); valid recall is 0.879 vs 0.864 (no-CF vs CF). On a broader 114-pair slice not separated by the tested future-date, document-range, or year-absence rules, CF reduces invalid retention from 0.6704 to 0.2809 (−0.3896 [−0.4830, −0.2944]), with no resolved q+ recall difference (−0.0082 [−0.0487, 0.0294]). A validation-frozen year heuristic also performs much worse than ConSieve on strict pair discrimination among 171 both-year pairs (0.210 vs 0.959). The tested date cues thus do not explain the full rejection effect.

Evidence redirection beyond all-PRUNE negatives

W1: The main diagnostic also makes entire documents unanswerable, where the target is to prune everything, rather than selecting among competing evidence.

On 255 same-document pairs where both queries are answerable, full-data ConSieve improves complete evidence switching over OpenProvence from 0.349 to 0.412 (+0.0627 [0.0076, 0.1179]). This is a full-model comparison, rather than a CF-specific causal attribution, and demonstrates evidence redirection beyond the all-PRUNE document-unanswerability setup.

Q1. Edit distribution and reproducibility

Among the 2,605 matched-training CF intervention slots (2,316 StreamingQA / 289 TimeQA), direct comparison of the actual query texts identifies 104 year-only substitutions, 2,242 other edits involving year/month tokens, 256 other textual edits, and 3 unchanged queries. These are observed surface-text categories rather than generator-assigned semantic edit labels. Separately, among 371 StreamingQA diagnostic edits, 317 introduce a distinct year (66 year-only); of these, 55 introduce a future year, 188 fall outside the document's explicit year range, and 51 satisfy both cues.

We provide three actual training-set q+/q− pairs below. Each pair shares the same document and reference-date prefix (omitted for readability).

Edit typeOriginal query (q+)Counterfactual query (q−)
Acquisition yearWhat is the name of the Swedish company that Spyker bought in 2010?What is the name of the Swedish company that Spyker bought in 2008?
Appointment yearWho became the Home Secretary in 1965?Who became the Home Secretary in 1963?
Geographic entityWhat is the capital of Yunnan?What is the capital of Guizhou?

In the Spyker example, the unchanged 2011 document supports the 2010 acquisition of Saab, while 2008 also appears in the document, but only in vehicle-production statistics. The edited year thus precedes the reference date, falls within the document's stated year range, and is explicitly present in the text—yet does not support the requested acquisition. In the other examples, the documents associate the Home Secretary appointment with 1965 and identify Kunming as Yunnan's capital, respectively.

Q1: Which model generated the counterfactual queries and which model annotated evidence, for the training data and for the diagnostic?

qwen-max generates CF candidates and annotates training evidence; qwen-plus verifies diagnostic evidence and serves as the downstream generator/Judge. The generation template requests one answer-critical condition edit while preserving temporal granularity, and manual auditing confirmed 395/412 intended diagnostic support transitions (Appendix B.1).

W2/Q2. Matched-training QA at 256 tokens

Table 3 reports the frozen deployment evaluation, whereas the MAIN1500 rows here report a separate hosted evaluation of the matched-training checkpoints, with OpenProvence re-evaluated as the reference; the absolute scores therefore differ across the two evaluations. On the MAIN1500 common pairs, the matched CF arm improves F1 over no-CF (0.6362 vs 0.6249) and both exceed OpenProvence (0.6190).

SetModelF1 ↑EM ↑Judge ↑
MAIN1500OpenProvence0.61900.47360.6203
MAIN1500No-CF0.62490.47360.6390
MAIN1500CF0.63620.47960.6551
GT406OpenProvence0.55460.42120.5640
GT406ConSieve0.63480.48280.6355

Q2. Non-temporal matched ablation

The held-out human-written SQuAD diagnostic tests non-temporal transfer without SQuAD fine-tuning. Matched CF improves entity valid recall and invalid rejection; numeric edits improve rejection at a recall cost, while negation worsens. The overall invalid-retention reduction is modest, so we characterize transfer as family dependent.

SQuAD edit familyValid recall, no-CF → CFInvalid retention, no-CF → CF
Overall0.942 → 0.9540.635 → 0.625
Entity0.948 → 0.9600.541 → 0.514
Antonym0.948 → 0.9560.696 → 0.687
Negation0.921 → 0.9450.785 → 0.831
Non-temporal number0.945 → 0.9270.482 → 0.378

Q3. Explaining budget effects and testing ranking directly

Source composition explains the larger GT406 gains: TimeQA accounts for 50.7% of GT406 but 13.7% of MAIN1500. Reweighting MAIN1500 source effects to the GT406 mix yields +0.0776 F1, close to Figure 3's +0.0802. On MAIN1500, the F1 difference is +0.0039 [−0.0104, 0.0183] for 1,294 StreamingQA questions and +0.1490 [0.0922, 0.2054] for 206 TimeQA questions; the corresponding TimeQA EM and Judge gains are +0.1117 [0.0485, 0.1748] and +0.1311 [0.0680, 0.1942].

To test the requested rank-based mechanism, we use 95 diagnostic pairs (57 StreamingQA / 38 TimeQA) with the gold document in the original-query BM25 top-25 pool. Three date-balanced matched model pairs score the edited query against that unchanged pool and pack sentences under the paper's 256-token cap without a threshold. Means are averaged over three seeds; nominal 95% intervals use 10,000 document-cluster bootstrap draws. These models are separate from the Table 3 deployment checkpoint.

Reviewer_2CMjOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies whether RAG context compressors distinguish semantic relevance from evidence that satisfies a query’s answer-critical conditions. It introduces a fixed-evidence counterfactual diagnostic: one query condition is changed while the document and candidate sentences remain unchanged, making previously supporting evidence invalid. It then proposes ConSieve, a 315M-parameter ModernBERT extractive compressor trained on supported-query evidence labels and counterfactual queries with all-prune targets.

Validation includes 412 predominantly temporal diagnostic pairs, a matched supervision ablation, end-to-end QA with identical BM25 retrieval and context budgets, multi-budget evaluation, and human-written non-temporal transfer tests. ConSieve retains 12.8% of invalidated evidence, compared with 55.3%–72.1% for tested baselines. The matched ablation reduces invalid retention from 56.1% to 11.4%, supporting the contribution of counterfactual supervision. Under a shared 256-token cap, downstream QA F1 improves from the strongest baseline’s 0.608 to 0.631. Non-temporal transfer is mixed, with improvements for some edit families but no overall advantage over OpenProvence. The main contribution is a controlled diagnosis and targeted training recipe for constraint-sensitive evidence selection.

Critical strengths

Controlled experiments connect the training intervention to practical improvements. The matched ablation reduces invalid retention from 56.1% to 11.4%. Separately, end-to-end comparisons hold BM25 retrieval, the generator, and context budgets constant, improving QA F1 from 0.608 to 0.631 with a positive paired confidence interval. Together with the multi-budget analysis, these results substantiate the usefulness of the proposed supervision within the evaluated setting.

Critical weaknesses

W1. The training contribution is not isolated from ordinary answerability-negative training. In the main counterfactual setup, the edited query is unsupported by the entire document, and every supervised document token receives a PRUNE label. The matched ablation demonstrates that this supervision helps, but does not compare against similarly difficult, non-counterfactual unanswerable examples with matched PRUNE supervision. Although Appendix E additionally examines evidence switching, the main training comparison leaves unclear what condition-specific counterfactual editing contributes beyond learning to reject unsupported documents. This weakens the distinctiveness of the proposed training recipe.

W2. The demonstrated generalization is substantially narrower than the broader constraint-sensitive compression objective. On human-written non-temporal edits, ConSieve’s invalid retention is 55.7%, compared with OpenProvence’s 54.5% at equal supported-evidence recall, with no demonstrated overall advantage. Improvements for entity and number edits reverse for negation and antonym edits. Moreover, downstream QA evaluation remains within StreamingQA/TimeQA, the same dataset families used for training. Although the paper acknowledges this limitation, the evidence primarily supports targeted temporal adaptation rather than a broadly transferable improvement in RAG evidence selection, limiting the contribution’s general significance.

Questions for authors

Q1. In the unmodified MAIN1500 and GT406 queries with actual BM25@25 results, how frequently does topically relevant but constraint-invalid evidence occur, and how often do compressors retain it? The fixed-evidence diagnostic deliberately constructs unsupported query edits; what evidence establishes that these cases represent a substantial failure mode in ordinary RAG?

Q2. Beyond the matched training ablation, which analyses connect improved QA answers specifically to removing constraint-invalid evidence? On the same evaluation set, are gains concentrated in examples containing such evidence, and how many errors are corrected or introduced by its removal? Can the benefit be distinguished from generic unsupported-document rejection or more conservative pruning?

Q3. What existing results establish transfer beyond the predominantly temporal StreamingQA/TimeQA setting? The human-written non-temporal diagnostic shows no overall advantage over OpenProvence, despite improvements for some edit families. Is there independent evidence of transfer across domains or constraint types, and does it require target-domain threshold calibration? Please distinguish the generalization claims supported by current experiments from those that remain untested.

1 comment

Dear Reviewer 2CMj,

Thank you for identifying the need to isolate the training intervention and connect the diagnostic to budgeted RAG. We address both directly.

W1. Condition editing versus generic unanswerability training

W1: The matched ablation demonstrates that this supervision helps, but does not compare against similarly difficult, non-counterfactual unanswerable examples with matched PRUNE supervision.

We added a 98-anchor, human-reviewed CF/generic-negative control (71 StreamingQA / 27 TimeQA). The two freshly trained arms share OpenProvence initialization, 19,985 common examples, negative documents, document-level PRUNE targets, source weights, sample order, and a 660-update schedule. The negative query alone changes: generic negatives request another unavailable fact about the same subject, whereas CF edits an answer-critical temporal condition. Query length and frozen OpenProvence scores are approximately matched; checkpoint and threshold procedures are shared.

98-anchor control, evaluated on 412 diagnostic pairsThresholdValid recall ↑Invalid retention ↓AUROC ↑Pair discrimination ↑
Generic unanswerable0.35000.89760.47730.80030.9583
Condition-edited CF0.45000.91000.33320.85990.9789

CF reduces invalid retention by 30.2% relative (−0.1441 [−0.1859, −0.1030]) and raises AUROC by +0.0596 [0.0455, 0.0737]. At a common 0.40 threshold, the gains persist in both valid recall (0.9165 vs 0.8851; +0.0314 [0.0144, 0.0483]) and invalid retention (0.3773 vs 0.4448; difference −0.0675 [−0.1078, −0.0282]). Separately, the larger matched CF/no-CF ablation reduces invalid retention from 0.5609 to 0.1141. These controls distinguish targeted condition editing from both in-domain adaptation and the tested generic-unanswerable supervision.

W2/Q3. Held-out non-temporal transfer and threshold calibration

W2: The demonstrated generalization is substantially narrower than the broader constraint-sensitive compression objective.

Q3: Is there independent evidence of transfer across domains or constraint types, and does it require target-domain threshold calibration?

On 3,315 paragraph-disjoint, human-written SQuAD pairs not used for fine-tuning, matched CF improves entity valid recall (0.948→0.960), reduces entity invalid retention (0.541→0.514; reduction 0.0268 [0.0095, 0.0445]), and increases pooled AUROC by +0.0101 [0.0068, 0.0132]. Importantly, transferring the temporal-validation thresholds without target-domain recalibration still reduces invalid retention by 0.0395 [0.0293, 0.0498], without a resolved valid-recall difference. These results support transfer to selected non-temporal constraint families, not a uniform improvement: negation worsens, and the full-data overall ConSieve–OpenProvence retention difference remains inconclusive (+0.0112 [−0.0048, 0.0268]).

Q1/Q2. Natural-retrieval relevance and budgeted evidence selection

In 95 fixed-pool diagnostic pairs (57 StreamingQA / 38 TimeQA), we evaluate edited queries on their original queries' BM25 top-25 pools, with three date-balanced matched model pairs, the paper's 256-token rank-based packing, and its qwen-plus reader prompt. CF reduces old-evidence retention from 0.587 to 0.389 (−0.198 [−0.279, −0.120]) and responses matching the original answer from 0.474 to 0.319 (−0.154 [−0.226, −0.087]). This directly demonstrates query-conditioned old-evidence suppression under cross-document budget competition, rather than only under a validation-selected pruning threshold.

The failure mode also occurs in ordinary retrieval: explicit constraint conflicts appear in reviewed sentences for at least 27/60 audited MAIN1500 queries.

Reviewer_YPGLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether extractive RAG compressors keep evidence that is still on topic but no longer valid after one query condition changes, mostly a date. The authors build a 412-pair same-document diagnostic from StreamingQA and TimeQA. They then propose ConSieve, which is OpenProvence fine-tuned on the same datasets plus LLM-generated counterfactual queries labeled as fully pruned. ConSieve keeps much less invalidated evidence on the diagnostic and gives a small F1 gain in BM25 RAG at a 256-token budget.

Critical strengths

The fixed-evidence diagnostic is a clean way to isolate how a selector reacts to a validity change, and the matched ablation is carefully controlled. The authors also report their negative results openly, including the SQuAD transfer and the per-dataset breakdown.

Critical weaknesses
  1. Most of the gain seems to come from in-domain fine-tuning rather than counterfactual supervision. ConSieve is trained on the same distribution as all test sets, while every baseline is used off the shelf. The no-CF arm already matches ConSieve on q+ F1 (0.743 vs 0.744) and reaches 0.606 F1 on GT406, well above OpenProvence in Figure 3. The extra gain from counterfactuals is about 0.016 F1, from a single seed with no interval.

  2. The learned behavior does not generalize. On human-written SQuAD edits, CF vs no-CF invalid retention is 0.625 vs 0.635, and ConSieve vs OpenProvence is 0.557 vs 0.545, both with intervals that include zero. ConSieve is clearly worse on negation and antonym edits. Even on TimeQA, which needs real interval reasoning, its invalid retention (0.445) is worse than EXIT and Sentinel. The strong numbers come almost entirely from StreamingQA, which looks more like a date-mismatch shortcut than general constraint sensitivity.

  3. The diagnostic does not match how the model is used. The diagnostic measures thresholded pruning, but under the 256-token cap ConSieve drops the threshold and just fills the budget by rank (Sec. 4.1). The motivating failure, wrong answers on edited or unanswerable queries, is never tested end to end. The downstream gain over OpenProvence is small (+0.024 F1, and the Judge interval includes zero). At natural operating points ConSieve actually keeps more text than OpenProvence (1,322 vs 848 tokens).

  4. Novelty is limited. Generating same-document unanswerable queries as hard negatives is essentially DocReRank (EMNLP 2025). Using unanswerable counterfactuals to align relevance with answerability was done in PiCL (Song et al., 2023), which is not cited. Single-condition counterfactual negatives are formalized in CausalNeg, and temporal contrast-set diagnostics for RAG already exist in TempRAGEval (EMNLP Findings 2025). The method itself is a data augmentation recipe on an existing pruner.

Questions for authors
  1. How do OpenProvence and EXIT perform after fine-tuning on the same StreamingQA/TimeQA data? Please add the no-CF arm to Tables 1 and 3.
  2. Would generic negatives, such as the original query paired with an unrelated document, give the same effect as counterfactual queries?
  3. Were the diagnostic edits produced by the same LLM and prompt as the training counterfactuals? How does a simple year-matching heuristic do on the diagnostic?
  4. Will code, checkpoints and the diagnostic set be released? None is linked in the submission.
3 comments

Dear Reviewer YPGL,

Thank you for your careful review. We address the concerns and questions below.

W1/Q1. Separating counterfactual supervision from in-domain adaptation

W1: Most of the gain seems to come from in-domain fine-tuning rather than counterfactual supervision.

The matched CF/no-CF ablation holds OpenProvence initialization, source examples, weighting, order, training schedule, and checkpoint selection fixed. Only 2,605 of 22,590 example positions change (2,602 with distinct queries): the CF arm replaces the supported query and its labels with a condition-edited query and document-level PRUNE supervision. CF reduces invalid-evidence retention from 0.5609 to 0.1141 (79.7% relative; difference −0.4468, nominal 95% CI [−0.4934, −0.4000]) and increases AUROC from 0.7954 to 0.9449 (+0.1494 [0.1297, 0.1676]), with valid recall 0.9053→0.8926. The key CF contribution is therefore query-conditioned rejection of invalidated evidence, beyond extraction gains obtainable through in-domain adaptation.

Model (412-pair diagnostic)q+ F1 ↑Valid recall ↑Invalid retention ↓AUROC ↑
OpenProvence0.66830.85370.72100.6257
Full-data ConSieve0.74450.89850.12760.9420
Matched no-CF0.74340.90530.56090.7954
Matched CF0.72800.89260.11410.9449

On MAIN1500 at 256 tokens, the matched no-CF arm reaches 0.6249 F1 and 0.6390 Judge, compared with 0.6190 and 0.6203 for OpenProvence. Our additional generic-negative control below isolates condition editing from ordinary unanswerability supervision.

W2. TimeQA and non-temporal transfer

W2: Even on TimeQA, which needs real interval reasoning, its invalid retention (0.445) is worse than EXIT and Sentinel.

On the 41 TimeQA diagnostic pairs, ConSieve improves valid recall (0.752 vs 0.713), invalid retention (0.445 vs 0.602; lower is better), and AUROC (0.700 vs 0.621) over OpenProvence. EXIT and Sentinel attain lower invalid retention on this subset only at substantially lower valid recall.

W2: The learned behavior does not generalize.

On 3,315 paragraph-disjoint, human-written SQuAD pairs, evaluated at matched valid recall (~0.943), ConSieve reduces invalid retention by 0.0736 [0.0511, 0.0958] on entity edits and 0.1646 [0.1006, 0.2317] on non-temporal number edits. SQuAD is not used in additional fine-tuning. Transfer is constraint-family dependent: the aggregate difference is inconclusive (+0.0112 [−0.0048, 0.0268]), and antonym/negation edits favor OpenProvence.

W3. Practical QA under the same budget

W3: At natural operating points ConSieve actually keeps more text than OpenProvence (1,322 vs 848 tokens).

With identical BM25@25 retrieval, reader, and a 256-token per-query cap, the deployment ConSieve checkpoint improves MAIN1500 F1 over OpenProvence by +0.0239 [0.0093, 0.0384] (Table 3). Retained context lengths are comparable (226.2 vs 224.9 tokens), as are model scale (~315M parameters) and median pruning latency (3.27 vs 3.25 s). The shared-cap comparison controls for the markedly different natural-threshold context lengths (1,322 vs 848 tokens). ConSieve also has the highest F1 point estimate at all five GT406 budgets (Figure 3). Under the hard cap, sentences are packed by score rather than thresholded; the controlled cross-document packing analysis described to Reviewers 2CMj and GKSD additionally tests condition-sensitive selection under this policy.

W4/Q2. Contribution beyond prior hard negatives

DocReRank, PiCL, CausalNeg, and TempRAGEval study related negatives or temporal contrasts (we will add the PiCL citation). Our contribution is the fixed-evidence diagnostic for post-retrieval sentence selection, together with a targeted supervision test and matched-budget RAG evaluation.

Q2: Would generic negatives, such as the original query paired with an unrelated document, give the same effect as counterfactual queries?

To distinguish condition edits from generic unanswerability training, we trained two arms with 98 human-reviewed anchors (71 StreamingQA / 27 TimeQA), keeping negative documents, PRUNE labels, initialization, common examples, source weights, example order, and the 660-update schedule fixed. The negative query construction is the intervention; lengths and frozen OpenProvence scores are approximately matched.

Compared with generic same-subject unanswerable queries, condition-edited CF reduces invalid retention from 0.4773 to 0.3332 (−0.1441 [−0.1859, −0.1030]), raises valid recall from 0.8976 to 0.9100, and raises AUROC by 0.0596 [0.0455, 0.0737]. Crucially, at the same threshold (0.40), CF simultaneously improves valid recall (0.9165 vs 0.8851) and invalid retention (0.3773 vs 0.4448). Thus, condition editing contributes selectivity beyond the tested generic-unanswerable recipe.

Q3/Q4. Construction, heuristic baseline, and release

Q3: Were the diagnostic edits produced by the same LLM and prompt as the training counterfactuals? How does a simple year-matching heuristic do on the diagnostic?

The counterfactual template requests one answer-critical condition edit while preserving temporal granularity. qwen-max generates counterfactual candidates and annotates training evidence; qwen-plus (temperature 0) verifies diagnostic evidence. Training and diagnostic counterfactual generation use the same settings. A manual audit confirmed 395/412 intended support transitions (Appendix B.1). A validation-frozen year heuristic achieves strict pair discrimination 0.543 vs 0.979 for ConSieve overall, and 0.210 vs 0.959 on 171 pairs where both queries contain years.

Q4: Will code, checkpoints and the diagnostic set be released?

We plan to release training/evaluation code, checkpoints, diagnostic data, and cached QA outputs.

I appreciate the authors' detailed response and the additional experiments. The supplementary results for "matched no-CF" on the MAIN1500 dataset, the "generic-negative" control experiments, and the year-based heuristic baseline have addressed most of my concerns. I maintain my original positive rating and hope that the authors will include these results in the final version, add a citation for PiCL, and—as promised—make the code and data publicly available.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Rebuttal Summary

We sincerely thank all reviewers and the chairs for their time and constructive feedback. We would like to summarize our main contributions, experimental results, and how we addressed the reviewers' concerns through extensive additional experiments.

1. Methodological Novelty and Contributions

ConSieve addresses an important problem in RAG compression: evidence can be semantically relevant to a query but fail to satisfy its answer-critical conditions. Our main contributions are:

  • Controlled Diagnostic: We introduce a fixed-evidence counterfactual diagnostic that directly measures whether compressors recognize when previously supporting evidence becomes invalid.
  • Targeted Training: We propose counterfactual supervision that teaches a lightweight compressor to reject invalid evidence. Matched ablations confirm the contribution of this training strategy.
  • Practical RAG Improvements: We demonstrate improvements in downstream QA under identical retrieval and context budgets, supported by additional generalization and evidence-switching experiments.

Our novelty lies in identifying a distinct failure mode, developing a targeted solution, and validating its effectiveness.

2. Extensive Experimental Evaluation

We conducted comprehensive experiments covering evidence selection, downstream QA, generalization, and efficiency:

  • 412 diagnostic pairs: ConSieve reduces invalid retention to 12.8% versus 72.1% for the baseline model while improving valid-evidence selection.
  • Matched ablation: Counterfactual supervision reduces invalid retention from 56.1% to 11.4%.
  • 1,500 RAG queries: Under identical retrieval, generation, and a 256-token cap, ConSieve achieves 0.631 F1 versus 0.608 for the strongest baseline.
  • Additional evaluations: Five context budgets, 3,315 human-written non-temporal pairs, 255 evidence-switching pairs, and efficiency comparisons.

3. Comprehensive Response to Reviewer Concerns

During the rebuttal, we carefully addressed the reviewers' concerns with substantial new experiments rather than explanations alone:

  • Novelty beyond generic negatives: A controlled 98-anchor experiment demonstrates that targeted counterfactual supervision reduces invalid retention from 47.7% to 33.3% compared with generic unanswerable training.
  • Shortcut learning: New shortcut-resistant tests show that ConSieve's improvements persist even when simple year-based cues are removed, directly addressing concerns about superficial pattern matching.
  • Beyond all-PRUNE behavior: On 255 same-document pairs where both queries are answerable, ConSieve improves complete evidence switching from 34.9% to 41.2%, demonstrating its ability to select different evidence for different queries.
  • Realistic RAG settings: Additional BM25 retrieval and hard-budget experiments confirm improvements under competing evidence and practical context constraints.
  • Training contribution: New matched no-CF experiments further connect counterfactual supervision to downstream QA improvements.
  • Generalization: Additional human-written experiments confirm transfer to selected non-temporal constraints while helping us clearly identify the remaining limitations.

We believe these new experiments have directly addressed and substantially resolved the major technical concerns raised during review, particularly regarding novelty, shortcut learning, the training mechanism, and practical effectiveness.

4. Reviewer Feedback After Rebuttal

We appreciate the reviewers' careful assessments and constructive suggestions:

  • Reviewer VPGL explicitly acknowledged that our new matched no-CF, generic-negative, and heuristic experiments addressed most concerns and maintained a positive rating.
  • Reviewers 2CMJ and GKSD: We provided detailed point-by-point responses and substantial new experimental evidence addressing their major concerns. Although we have not received further feedback from these reviewers, we believe our responses have substantially strengthened the paper and hope the additional results will be carefully considered in the final assessment.

We believe ConSieve makes a meaningful contribution through a novel diagnostic, targeted counterfactual training, and demonstrated improvements in practical RAG. The extensive additional experiments have strengthened our findings and addressed the central concerns raised during review.

Thank you again for your time and thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.