acceptodds
Under review as a conference paper at ICLR 2027

Closing the Gap: A Two-Stage Training Recipe for Soft Context Compression in RAG

Abstract

Retrieval-Augmented Generation (RAG) improves knowledge-intensive generation by conditioning language models on retrieved documents, but processing these documents becomes increasingly expensive as retrieval depth grows. Soft context compression reduces this cost by encoding documents into compact continuous representations that can be precomputed and reused across queries. However, many existing methods train compressed models by distilling from a full-context teacher. When the teacher is wrong, such distillation can reinforce its errors, while teacher imitation provides no direct signal for improving beyond the teacher. We propose DEX-Comp, a two-stage training recipe that separates reliable imitation from targeted exploration. Pure Distillation learns only from teacher-correct questions to mitigate error propagation, while Hard Exploration applies outcome-based reinforcement learning to teacher-failed questions to directly optimize answer correctness. Across five open-domain QA benchmarks and retrieval depths from top- to top-, DEX-Comp at compression outperforms all evaluated compression baselines and surpasses the untuned full-context RAG model in average accuracy, while reducing time-to-first-token by –. Evaluations across additional datasets and backbones further demonstrate its generalization.

Primary area:
generative models
Keywords:
context compression, RAG

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BVw3OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

DEX-Comp is a two-stage training recipe for query-independent soft context compression in RAG. A compressor maps each retrieved document into a shorter sequence of continuous embeddings, which a compression-aware decoder consumes. The key idea is to allocate different learning signals according to teacher correctness. Stage I, Pure Distillation, trains only on teacher-correct examples, aligning the compressed student’s response distribution with a frozen full-context RAG teacher via KL divergence. Stage II, Hard Exploration, initializes from the PD checkpoint and trains on teacher-failed examples with outcome-based GRPO, directly rewarding answer correctness rather than teacher agreement.

The paper evaluates on five open-domain QA datasets at retrieval depths top-5, top-15, and top-30, using Mistral-7B and several query-independent compression baselines. At 16× compression, DEX-Comp surpasses uncompressed RAG on average CEM and LLM-judged accuracy across depths, and substantially improves TTFT, GFLOPs, and peak memory. Ablations support the PD/HE split, and additional experiments show generalization across backbones and OOD datasets.

Critical strengths

1.The teacher-correctness-based allocation of distillation and exploration is well motivated and empirically supported. PD trains only on S+\mathcal{S}^{+} , avoiding imitation of teacher-failed responses, while HE uses S−\mathcal{S}^{-} with outcome-based GRPO. The ablation shows PD improves average LLM-judged accuracy from 64.79 to 67.75 before RL, and HE further improves it to 72.02. Exploration on teacher-failed examples also outperforms exploration on teacher-correct examples (70.60) or their mixture (71.47) from the same PD checkpoint. This is consequential because it directly validates the paper’s central design choice, not just the overall pipeline.

2.DEX-Comp closes the compression gap while delivering large efficiency gains. At 16× compression, it surpasses uncompressed RAG on every dataset and depth in the main table, with average CEM/LLM gains above two points. It also reduces TTFT by 4.4×–23.7×, GFLOPs by 6.67×–13.61×, and peak memory, with larger gains at greater retrieval depths. This is consequential because it shows compression is not merely an efficiency trade-off; it can match or exceed full-context RAG under realistic retrieval depths.

Critical weaknesses

The following are minor concerns rather than material weaknesses:

1.Teacher correctness is determined by Gemini 3 Flash. Judge errors could misallocate examples between (\mathcal{S}^{+}) and (\mathcal{S}^{-}), though the ablation and main results suggest the signal is sufficiently reliable.

2.OOD evaluation is mixed: DEX-Comp improves on CovidQA and FEVER but is lower on BioASQ. This is a limitation, not a contradiction of the main claims.

3.The “first query-independent soft compression method shown to surpass uncompressed RAG” claim is strong; the paper could phrase it more cautiously, but the evidence supports the narrower claim under its evaluated settings.

Questions for authors

1.Teacher-judge sensitivity: How sensitive are PD/HE partition and final accuracy to the choice of teacher-correctness judge? Can you report the fraction of examples whose correctness label changes under a different judge, and the downstream effect on DEX-Comp?

2.OOD BioASQ: Why does DEX-Comp underperform uncompressed RAG on BioASQ? Is this due to domain shift, retrieval depth, or compression rate? A short characterization would clarify when compression should be avoided.

3.Offline compression and storage cost: The paper notes that offline compression and embedding storage are amortized across queries. Can you quantify storage and recompression cost at 16× on the Wikipedia-KILT corpus, and how updated documents are handled?

2 comments
AuthorsOpenReview ↗

Response to Reviewer BVw3 (Part 1/2)

We sincerely thank you for your careful reading and positive assessment of our work. We also appreciate the valuable questions you raised, which motivated the further analyses presented below.

W1. Teacher correctness is determined by Gemini 3 Flash. Judge errors could misallocate examples between S+\mathcal{S}^{+} and S−\mathcal{S}^{-}, though the ablation and main results suggest the signal is sufficiently reliable.

Q1. Teacher-judge sensitivity: How sensitive are PD/HE partition and final accuracy to the choice of teacher-correctness judge? Can you report the fraction of examples whose correctness label changes under a different judge, and the downstream effect on DEX-Comp?

We thank you for this helpful suggestion. To investigate the effect of judge choice, we replace the original Gemini 3 Flash judge with Gemini 3.8 Flash. As a result, both the PD/HE partition and final accuracy remain stable under this change.

Each judge partitions the training questions into teacher-correct examples for Pure Distillation (PD) and teacher-failed examples for Hard Exploration (HE). Label changes are measured against the original Gemini 3 Flash partition. All final accuracies are evaluated by Gemini 3 Flash, reporting the five-dataset mean at top-30 retrieval and 16× compression.

Teacher-correctness judgePD / teacher-correct (%)HE / teacher-failed (%)Label changes (%)Final avg. accuracy (%)
Gemini 3 Flash (original)67.2532.75–72.02
Gemini 3.8 Flash68.5531.452.172.14

Changing the judge alters only 2.1% of labels, with a final accuracy difference of just 0.12 points. These results support the robustness of both the data partition and final performance to the choice of judge model.

W2. OOD evaluation is mixed: DEX-Comp improves on CovidQA and FEVER but is lower on BioASQ. This is a limitation, not a contradiction of the main claims.

Q2. OOD BioASQ: Why does DEX-Comp underperform uncompressed RAG on BioASQ? Is this due to domain shift, retrieval depth, or compression rate? A short characterization would clarify when compression should be avoided.

We thank you for this helpful question. To investigate the BioASQ gap, we inspected 50 questions that DEX-Comp answered incorrectly. The observed failure pattern points primarily to domain shift, particularly on factoid questions requiring precise biomedical terminology. The errors include substitutions between similar names or related concepts, as illustrated below.

QuestionReference answerDEX-Comp answer
Which library is used for fixed-length approximate string matching?libFLASMlibFLASH
Which class of disorders are caused by AMPA receptor GluA2 subunit defects?Neurodevelopmental disordersNeurodegenerative disorders
Which microRNAs are involved in targeting CYLD in triple negative breast cancer?miR-182miR-181b

This pattern is consistent with a mismatch between BioASQ's specialized vocabulary and our general-domain, Wikipedia-based training, which may make precise biomedical distinctions harder to preserve under compression. We therefore recommend caution with soft compression in domains rich in rare terminology, especially when exact names, numerical values, or sequence motifs determine correctness.

AuthorsOpenReview ↗

Response to Reviewer BVw3 (Part 2/2)

W3. The “first query-independent soft compression method shown to surpass uncompressed RAG” claim is strong; the paper could phrase it more cautiously, but the evidence supports the narrower claim under its evaluated settings.

We thank you for this helpful reminder. This “first” claim appears only in Section 5.1 (Main Results), where we discuss the experimental results. We will revise it to a more cautious statement limited to our evaluated settings.

Q3. Offline compression and storage cost: The paper notes that offline compression and embedding storage are amortized across queries. Can you quantify storage and recompression cost at 16× on the Wikipedia-KILT corpus, and how updated documents are handled?

We thank you for this practical question. At 16× compression, caching the BF16 embeddings for approximately 23.85M Wikipedia-KILT documents requires about 1.56 TB. A full-corpus compression pass will take only 6.9 hours on eight RTX PRO 6000 Blackwell GPUs. For subsequent updates, we compress only new or modified documents. With a single-GPU compression throughput of 119.8 documents/s, this keeps small updates inexpensive. We then insert or replace the corresponding embeddings in the document store. When downstream queries retrieve these documents, the generator uses the updated cached embeddings in place of the original document text.

Overall, offline compression trades storage for inference time: for a fixed compressor, the full corpus is processed once, while later updates are incremental. The more often documents are reused across queries, the greater the cumulative inference time saved and the more the initial compression cost is amortized.

Reviewer_uSAoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the cost of processing retrieved documents in RAG through query-independent soft context compression. Documents are encoded into compact continuous representations that can be precomputed and reused across queries. DEX-Comp uses two training stages: Pure Distillation learns from questions answered correctly by a full-context teacher, while Hard Exploration applies GRPO to questions the teacher answers incorrectly, using answer correctness as the reward. The aim is to avoid copying teacher errors while allowing the compressed model to improve beyond imitation. Experiments cover five open-domain QA benchmarks and retrieval depths from top-5 to top-30. At 16x compression, the paper reports higher average accuracy than the evaluated compression baselines and the untuned full-context model, together with lower time-to-first-token and memory use. Ablations examine initialization and the choice of RL training subset.

Critical strengths

This is a simple training recipe with useful results. Table 3 checks both the initialization and the RL subset. At 16x compression, the method improves over the reported compression baselines, while the TTFT and memory measurements show a practical benefit. The full-context RL comparison is also valuable: it is more accurate at top-5 but costs more, which gives a fairer picture of the trade-off.

The accuracy and inference savings are useful in practice, even though the advantage of failure-only selection over mixed RL remains uncertain.

Critical weaknesses

Table 3 omits subset sizes and total rollout counts, making it difficult to isolate the benefit of selecting teacher-failure examples. With an epoch-based schedule, success-only, failure-only, and mixed training may use different budgets. The failure-only result is also fairly close to mixed RL, and only one training seed is specified. Please give the budgets used in these existing runs. If they differ, the paper should avoid attributing the small gap solely to the selection rule.

The short-answer reward also leaves a question about ASQA. RL uses answer containment and caps rollouts at 32 tokens, whereas ASQA is described as long-form QA. The reported scores support improved answer correctness, but do not by themselves establish complete, grounded long-form responses. The evaluation description should make that distinction clear.

Questions for authors
  1. What subset sizes, optimizer steps, and rollout counts were used for the existing Table 3 runs?
  2. How are ASQA answers generated and scored at evaluation, given the 32-token cap during RL?
3 comments
AuthorsOpenReview ↗

Response to Reviewer uSAo (Part 1/2)

We sincerely thank you for your valuable and insightful comments, which have greatly helped us improve our manuscript. Below, we provide detailed responses to each issue you raised.

W1. Table 3 omits subset sizes and total rollout counts, making it difficult to isolate the benefit of selecting teacher-failure examples. With an epoch-based schedule, success-only, failure-only, and mixed training may use different budgets. The failure-only result is also fairly close to mixed RL, and only one training seed is specified. Please give the budgets used in these existing runs. If they differ, the paper should avoid attributing the small gap solely to the selection rule.

Q1. What subset sizes, optimizer steps, and rollout counts were used for the existing Table 3 runs?

We sincerely thank you for your valuable feedback. RL on teacher-failed questions is an effective and efficient training choice: it achieves higher accuracy at lower cost in the original runs, and remains the best choice when the training budget is held fixed. Below, we report the original budgets and a new controlled comparison addressing your concern.

  1. Training budgets for the existing runs. The table below supplements the RL-subset comparison in Table 3 with training budgets and significance analysis. All three variants use top-30 retrieved documents and start from the same Stage-I Pure Distillation checkpoint, trained by imitating correct teacher responses. Success-only uses teacher-correct questions, Failure-only uses teacher-failed questions, and Mixed uses both. Each optimizer step processes 64 questions per GPU across eight GPUs, with 16 responses sampled per question. Accuracy is the five-dataset mean LLM-judged score (%), and gains and 95% confidence intervals (CIs) are in percentage points.

    RL subsetQuestionsOptimizer stepsRolloutsGPU-hoursAvg. Acc.Gain of Failure-only (p; 95% CI)
    Success-only304,6478006.55M5370.60+1.42 (p < 0.001; [+0.92, +1.89])
    Mixed (full pool)453,0231,2009.83M7971.47+0.55 (p < 0.001; [+0.26, +0.84])
    Failure-only (ours)148,3764003.28M2672.02–

    Failure-only achieves the highest accuracy with approximately 67% fewer GPU-hours than Mixed, and the gain is statistically significant under a paired permutation test. These results support Failure-only as an effective and efficient training choice. We will report the training budgets in the revised appendix and clarify that, because these budgets differ, the accuracy gain cannot be attributed solely to data selection.

  2. Higher teacher-failed proportions improve accuracy at a fixed training budget. To separate data composition from training budget, we hold the training-pool size at 148,376 questions, the optimizer steps at 400, and total rollouts at 3.28M, using the same Pure Distillation checkpoint and RL hyperparameters. We retain the original Failure-only pool, sample an equally sized pool of teacher-correct questions for Success-only, and construct a Balanced Mixed pool with 74,188 questions from each subset.

    RL subsetTeacher-failed proportionQuestionsOptimizer stepsRolloutsAvg. Acc. (%)
    Success-only0%148,3764003.28M69.08
    Balanced Mixed50%148,3764003.28M71.12
    Failure-only (ours)100%148,3764003.28M72.02

    With these budgets matched, accuracy increases as the teacher-failed proportion rises across the three settings. Failure-only exceeds Balanced Mixed by 0.90 points and Success-only by 2.94 points, supporting teacher-failed questions as a more effective source of RL training signal under the same budget.

AuthorsOpenReview ↗

Response to Reviewer uSAo (Part 2/2)

W2. The short-answer reward also leaves a question about ASQA. RL uses answer containment and caps rollouts at 32 tokens, whereas ASQA is described as long-form QA. The reported scores support improved answer correctness, but do not by themselves establish complete, grounded long-form responses. The evaluation description should make that distinction clear.

Q2. How are ASQA answers generated and scored at evaluation, given the 32-token cap during RL?

We thank you for pointing out this important detail. The apparent mismatch arises from the difference between ASQA's original format and the processed version used in our experiments. For both training and evaluation, we directly reuse PISCO's data-processing setup [1], using BERGEN [2] as already stated in Section 4.3 (Implementation Details).

ASQA annotates each ambiguous question with multiple short answers corresponding to its different interpretations, alongside human-written long-form answers covering these interpretations [3]. BERGEN removes the long-form annotations and merges the short answers into a single reference list [4]. Our reward and CEM metric use the same matching rule as BERGEN: a response is correct if it contains any one of these short answers after normalization. The LLM judge evaluates correctness against the same reference list.

The preprocessed version of ASQA we reuse from BERGEN therefore differs from the original long-form QA dataset. We will clarify this distinction in the revised manuscript to avoid confusion.

 

References

[1] Maxime Louis et al., PISCO: Pretty Simple Compression for Retrieval-Augmented Generation, ACL 2025.

[2] David Rau et al., BERGEN: A benchmarking library for retrieval-augmented generation, EMNLP 2024.

[3] Ivan Stelmakh et al., ASQA: Factoid Questions Meet Long-Form Answers, EMNLP 2022.

[4] BERGEN implementation: ASQA preprocessing and match-score computation.

AuthorsOpenReview ↗

Follow-up on Our Response and Revisions

Dear Reviewer uSAo,

As today is the final day of the discussion period, we would like to follow up on our response. We have carefully considered your concerns, provided detailed replies, and revised the manuscript based on your suggestions. The corresponding changes are marked in orange.

We sincerely appreciate your time, effort, and valuable feedback. We would be grateful to hear whether our responses and revisions have addressed your concerns.

Best regards,

The authors

Reviewer_5Y4BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RAG improves knowledge-intensive generation by conditioning language models on retrieved documents, but processing these documents becomes increasingly expensive as retrieval depth grows. Soft context compression reduces this cost by encoding documents into compact continuous representations that can be precomputed and reused across queries. However, standard distillation from a full-context teacher can reinforce teacher errors while providing no direct signal for improving beyond the teacher. The authors propose DEX-Comp, a two-stage training recipe: Pure Distillation learns only from teacher-correct questions to mitigate error propagation, while Hard Exploration applies outcome-based reinforcement learning to teacher-failed questions to directly optimize answer correctness. Across five open-domain QA benchmarks and retrieval depths from top-5 to top-30, DEX-Comp at 16x compression outperforms evaluated compression baselines and surpasses the untuned full-context model in average accuracy, while reducing time-to-first-token by 4.4x to 23.7x

Critical strengths
  • The motivation is concrete and well diagnosed. Figure 1(b) shows that a substantial fraction of the distilled model's errors match those of the teacher, which supports the claim that standard distillation imitates teacher errors regardless of whether the teacher is correct
  • Compression doubles as an efficient substrate for RL. At top-5 it supports a 4x larger per-GPU batch (64 vs 16) and uses 65.8% fewer GPU-hours (13 vs 38), and it remains feasible at top-30 where full-context RL exceeds the memory budget
  • Across the five open-domain QA benchmarks at retrieval depths from top-5 to top-30, DEX-Comp at 16x compression consistently outperforms all evaluated compression baselines and the untuned full-context model
  • Measured inference acceleration is substantial, cutting time-to-first-token by 4.42x at top-5 to 23.73x at top-30 while reducing peak GPU memory by up to 4.02x (Table 2)
Critical weaknesses
  • The claim of surpassing full-context RAG relies on an untuned baseline. When the full-context model is also trained with RL, it reaches 76.78% at top-5, beating DEX-Comp's 72.54% by over four points (Table 4). So the result is an efficiency improvement, not an accuracy gain over an equally trained full-context system
  • DEX-Comp uses a gold-answer signal through both the correctness filter and the CEM reward, while distillation baselines like PISCO and xRAG rely only on teacher outputs. The main table gives none of these baselines the same RL or gold-answer treatment, so the comparison mixes the two-stage recipe with the benefit of extra supervision. PISCO-plus-RL control in the main results would make the separation clean.
  • The central claim that exploration should run specifically on teacher-failed questions has weak support. In Table 3, RL on teacher failures (PD+Failure) reaches 72.02%, beating RL on a mixed set of successes and failures (PD+Mixed, 71.47%) by only 0.55 points, and the two are almost identical on TriviaQA (90.12% vs 90.10%). The paper reports no significance test on this, so the gap does not justify the added step of excluding teacher-correct questions from exploration since the mixed set uses more data and comes within noise
Questions for authors

Is PD+Failure (72.02) significantly better than PD+Mixed (71.47)? Without a significance test, what justifies excluding teacher-correct questions from exploration when the mixed set uses more data and is within noise?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 5Y4B(Part 1/3)

We sincerely thank you for your valuable and insightful comments, which have greatly helped us improve our manuscript. Below, we provide detailed responses to each concern you raised.

W2. DEX-Comp uses a gold-answer signal through both the correctness filter and the CEM reward, while distillation baselines like PISCO and xRAG rely only on teacher outputs. The main table gives none of these baselines the same RL or gold-answer treatment, so the comparison mixes the two-stage recipe with the benefit of extra supervision. PISCO-plus-RL control in the main results would make the separation clean.

We thank you for raising this important point, which we also considered when designing our experiments. In fact, the PISCO+RL control is already included in the original Table 3, and the corresponding analysis and conclusions are presented in Section 6.1. It is listed under “Standard distill.” with “Failure (HE).” PISCO’s core idea is to train the compression model through standard distillation alone [1], and we use exactly the same implementation for our “Standard distill.” baseline. Adding Hard Exploration (HE) therefore gives the PISCO+RL control. Both PISCO+RL and DEX-Comp apply RL to teacher-failed questions using the same gold-answer-based reward.

The following results are taken directly from the original Table 3: top-30 retrieval, 16× compression, and LLM-judged accuracy (%).

MethodNQTriviaQAHotpotQAASQAPopQAAvg.
PISCO+RL (Standard distillation + HE)69.4686.8351.8574.2249.6766.41
DEX-Comp (Pure Distillation + HE)75.5790.1258.4880.3855.5372.02

DEX-Comp outperforms PISCO+RL on all five datasets, with a +5.61 average gain, even when PISCO also receives RL and gold-answer rewards. This supports the value of our two-stage recipe beyond simply adding RL supervision. We will highlight this correspondence in Section 6.1 (Ablation Study) of the revised manuscript to make this control easier to identify.

AuthorsOpenReview ↗

Response to Reviewer 5Y4B(Part 2/3)

W1. The claim of surpassing full-context RAG relies on an untuned baseline. When the full-context model is also trained with RL, it reaches 76.78% at top-5, beating DEX-Comp's 72.54% by over four points (Table 4). So the result is an efficiency improvement, not an accuracy gain over an equally trained full-context system.

We thank you for raising this concern. Our claim throughout the paper is to close the gap between compressed and uncompressed RAG, with accuracy gains over the untuned full-context reference; this claim is directly supported by our experiments. Beyond outperforming all evaluated compression baselines and accelerating inference, DEX-Comp also enables performance improvements under a fixed training budget. Performance gains from training must be evaluated alongside the cost of achieving them. Final accuracy alone is insufficient when training budgets differ. We detail both points below.

  1. Our claim is to close the gap to untuned full-context RAG, and our experiments directly support it. We focus on offline, query-independent compression, where documents are encoded before the downstream query is known. Matching full-context accuracy is challenging because information needed by different future questions must fit into a much smaller representation [2]. Prior work—including xRAG [3], PISCO [1], and RL-Comp [4]—aims to match the accuracy of the untuned, uncompressed backbone, yet still reports gaps to this target. Despite incorporating RL, RL-Comp still treats this backbone as the “full-context upper bound” and remains 7.60 points below it in our main Table 1 (62.93 vs. 70.53 at top-5). Closing the gap to this untuned reference is therefore the immediate priority for offline compression, before pursuing the even higher accuracy of an additionally RL-trained full-context model.

    We examine this gap more closely using PISCO, the strongest compression baseline in our main table. All results below are taken directly from the original Tables 1 and 3: top-30 retrieval and LLM-judged accuracy (%). All compressed variants in this comparison use 16× compression.

    MethodUses RLNQTriviaQAHotpotQAASQAPopQAAvg.
    Untuned full-context RAGNo72.7588.2054.6476.7955.0469.49
    PISCONo67.5786.5950.2072.6846.8964.79
    PISCO+RL (same results as in W2)Yes69.4686.8351.8574.2249.6766.41
    Pure Distillation (PD)No70.9688.7455.3274.7948.9567.75
    DEX-Comp (PD + Hard Exploration)Yes75.5790.1258.4880.3855.5372.02

    Pure Distillation alone closes about 63% of the gap to untuned full-context RAG, while subsequent RL further improves accuracy, surpassing this reference on all five datasets. PD already outperforms PISCO+RL, showing that RL is not the sole source of improvement. Together with the main results across retrieval depths in Table 1, this establishes DEX-Comp as the strongest evaluated compression method and directly supports our stated claim of closing the gap and surpassing the untuned full-context reference.

  2. DEX-Comp enables effective and efficient training within a fixed budget. Table 4 and Section 6.2 show how compression makes RL more affordable. Compression thus enables performance improvements under limited training budgets, beyond its inference efficiency benefits. The table below supplements Table 4 with a top-5 full-context RL result at the same 13 GPU-hours as DEX-Comp (†).

    Retrieval depthMethodTraining GPU-hoursAvg. Acc. (%)
    Top-5Untuned full-context RAG–70.53
    Top-5Full-context + RL, matched budget (†)1372.62
    Top-5DEX-Comp1372.54
    Top-5Full-context + RL, original budget3876.78
    Top-30Untuned full-context RAG–69.49
    Top-30DEX-Comp2672.02
    Top-30Full-context + RLNot feasible: exceeds available memory–

    At top-30, compression makes RL feasible within our hardware budget, enabling accuracy gains over the untuned full-context baseline. Full-context RL exceeds our hardware's memory budget, whereas compressing first makes RL feasible and raises accuracy from the untuned reference's 69.49 to 72.02. At top-5, DEX-Comp achieves similar accuracy to full-context RL at the same 13 GPU-hours, with 4.42× faster online time-to-first-token. Full-context RL's higher final accuracy of 76.78 requires 2.92× the training GPU-hours. DEX-Comp therefore provides both efficient inference and an effective way to improve accuracy under training constraints.

AuthorsOpenReview ↗

Response to Reviewer 5Y4B(Part 3/3)

W3. The central claim that exploration should run specifically on teacher-failed questions has weak support. In Table 3, RL on teacher failures (PD+Failure) reaches 72.02%, beating RL on a mixed set of successes and failures (PD+Mixed, 71.47%) by only 0.55 points, and the two are almost identical on TriviaQA (90.12% vs 90.10%). The paper reports no significance test on this, so the gap does not justify the added step of excluding teacher-correct questions from exploration since the mixed set uses more data and comes within noise.

Q1. Is PD+Failure (72.02) significantly better than PD+Mixed (71.47)? Without a significance test, what justifies excluding teacher-correct questions from exploration when the mixed set uses more data and is within noise?

We sincerely thank you for your valuable feedback. We use only teacher-failed questions during RL because this is an effective and efficient training choice: it reuses the partition constructed for Stage-I Pure Distillation (PD) without additional filtering, saves approximately 67% of the GPU-hours relative to Mixed, and achieves higher accuracy (+0.55, p < 0.001). Below, we explain this rationale and provide additional evidence to address your concern.

  1. Failure-only achieves better accuracy at lower training cost. The table below supplements the RL-subset comparison in Table 3 with training budgets and significance analysis. All three variants use top-30 retrieved documents and start from the same Pure Distillation (PD) checkpoint with the same RL hyperparameters. S+ and S− denote teacher-correct and teacher-failed questions, respectively. Accuracy is the five-dataset mean LLM-judged score (%); gains and 95% confidence intervals (CIs) are in percentage points, and p-values are from paired permutation tests.

    RL subsetQuestionsRolloutsGPU-hoursAvg. Acc.Gain of Failure-only (p; 95% CI)
    Success-only (S+)304,6476.55M5370.60+1.42 (p < 0.001; [+0.92, +1.89])
    Mixed (S+ ∪ S−, full pool)453,0239.83M7971.47+0.55 (p < 0.001; [+0.26, +0.84])
    Failure-only (S−, ours)148,3763.28M2672.02–

    Thus, Failure-only achieves the highest accuracy with just 32.8% of the Mixed training pool, and its improvement over Mixed is statistically significant (p < 0.001).

  2. The Stage-I partition also reflects the student's remaining learning needs without extra screening. Since Stage I uses dense distillation to imitate correct teacher responses on S+, the PD student already receives strong supervision on this subset. Consistent with this, 42.0% of S+ questions are answered correctly in all 16 sampled responses, compared with only 5.6% of S− questions. This suggests more room for improvement on S− after distillation, motivating us to focus outcome-based exploration there. Moreover, reusing the Stage-I partition introduces no additional filtering cost. Repartitioning the full training pool by student pass@16 would instead require 7.25M additional rollouts, approximately 2.2× the Failure-only RL rollout budget.

For TriviaQA, the two choices indeed perform almost identically (90.12 vs. 90.10). We see two plausible reasons: first, PD already achieves 88.74% accuracy, suggesting limited headroom for further gains; second, the teacher's high accuracy leaves relatively few failed cases for exploration, with TriviaQA contributing only 5.3% of the S− training pool. Both factors may limit the additional benefit of failure-focused RL.

We will incorporate this rationale for choosing Failure-only, including the significance analysis you requested, into the revised manuscript.

 

References

[1] Maxime Louis et al., PISCO: Pretty Simple Compression for Retrieval-Augmented Generation, ACL 2025.

[2] Maxime Louis et al., OSCAR: Online Soft Compression And Reranking, ICLR 2026.

[3] Xin Cheng et al., xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token, NeurIPS 2024.

[4] Yujan Ting et al., Bridging the Memorization-Utilization Gap: Near-Lossless Context Compression via Reinforcement Learning, ACL 2026.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Manuscript Update

Dear Reviewers,

We sincerely thank you for your thoughtful and valuable feedback, which has greatly helped us improve the quality and clarity of our manuscript. We have carefully considered your concerns, provided detailed responses and additional results, and updated the manuscript accordingly. For ease of reference, revisions addressing your comments are marked in blue for 5Y4B, orange for uSAo, and green for BVw3.

We would be grateful if you could review our responses and the revised manuscript and let us know whether they address your concerns. We look forward to further discussion and would be happy to answer any remaining questions.

Thank you again for your time and consideration.

Best regards,
The authors

Dear Reviewers, ACs, and SACs,

We sincerely thank all reviewers for their thorough and constructive feedback. We have not yet received responses to our rebuttal, which we fully understand given how busy this period is. We would be sincerely grateful if the reviewers could find a moment to read our responses and revisions, which we believe address the concerns they raised.

In response to the reviewers' valuable suggestions, we have updated the manuscript with additional experiments and clarifications, as detailed below.

Paper Summary

DEX-Comp revisits the underexplored role of training in offline soft context compression. We observe teacher-error propagation in standard distillation and address it with a targeted two-stage recipe: Pure Distillation (PD) learns only from teacher-correct questions, then Hard Exploration (HE) uses teacher-failed questions to improve beyond imitation. Changing the training recipe alone yields substantial gains: at 16× compression, DEX-Comp surpasses all evaluated compression baselines and the untuned full-context reference in average QA accuracy. We believe this simple and effective approach provides a step toward understanding and improving soft context compression.

Strengths

We thank the reviewers for highlighting the key strengths of our work:

  • Clear motivation and a simple training recipe (Reviewers 5Y4B, uSAo, BVw3): 5Y4B described the motivation as “concrete and well diagnosed”; uSAo called the method “a simple training recipe with useful results”; and BVw3 found the allocation of distillation and exploration “well motivated and empirically supported.”
  • Strong accuracy and practical inference savings (Reviewers 5Y4B, uSAo, BVw3): All three reviewers recognized strong accuracy and efficiency gains: DEX-Comp surpasses the evaluated compression baselines and the untuned full-context reference in average accuracy, while reducing TTFT by 4.4×–23.7×.
  • Informative ablations (Reviewers uSAo, BVw3): Both reviewers recognized the value of our ablation study. BVw3 emphasized that it “directly validates the paper’s central design choice, not just the overall pipeline.”
  • Efficient RL training (Reviewers 5Y4B, uSAo): Both reviewers recognized that compression improves RL training efficiency. DEX-Comp lowers training costs and remains feasible at top-30, where full-context RL exceeds the available memory budget.

Response to Main Concerns

We have carefully addressed the main concerns raised by reviewers:

  • Failure-only exploration and training budgets (Reviewers 5Y4B, uSAo): We reported the original training budgets and added significance tests: Failure-only improves over Mixed by 0.55 points (p < 0.001) with approximately 67% fewer GPU-hours. A new comparison matching training budgets shows that Failure-only remains best. This supports Failure-only as an effective and efficient choice: it targets questions with more room for improvement after PD and reuses the Stage-I partition without additional filtering.
  • Contribution beyond additional RL supervision (Reviewer 5Y4B): We clarified that PISCO+RL is already included in the original Table 3. With the same teacher-failed RL data and gold-answer reward, DEX-Comp outperforms this control on all five datasets at top-30. This supports the value of the two-stage recipe beyond simply adding RL supervision.
  • Performance gains under limited resources (Reviewer 5Y4B): We added a comparison at equal training cost: DEX-Comp achieves similar accuracy to full-context RL at top-5 with the same 13 GPU-hours. At top-30, compression makes RL feasible within our hardware budget and improves accuracy over the untuned full-context reference. These results show that DEX-Comp closes the compression gap and provides an effective way to improve performance under limited resources.
  • ASQA preprocessing and scoring (Reviewer uSAo): We clarified that the apparent mismatch comes from preprocessing: both training and evaluation use PISCO's BERGEN version of ASQA, which removes long-form annotations and merges short answers into a reference list.

We also thank Reviewer BVw3 for giving our work the highest rating and finding no major weaknesses. We have also addressed their minor concerns about teacher-judge sensitivity, BioASQ, the “first” claim, and offline compression cost.

Revisions are marked in blue for 5Y4B, orange for uSAo, and green for BVw3.

Thank you again to all reviewers, the ACs, and the SACs for your time, effort, and constructive suggestions!

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.