acceptodds
Under review as a conference paper at ICLR 2027

DiffContext: Learning Adaptive Budgets for Risk-Controlled Context Compression

Abstract

Long contexts expand the information available to language models, but increase the cost of inference. Context compression offers a way to reduce this cost, yet its savings may come at the expense of evidence essential to the downstream task. The amount of context needed to preserve task performance varies across inputs: some contain repeated evidence, whereas others require complementary information from several passages. Applying the same retention budget across inputs can therefore discard necessary evidence in some cases while retaining unnecessary content in others. These differences motivate adaptive budget selection that balances token savings against the risk of task degradation. However, directly evaluating task quality at multiple budgets for each new input would require repeated task-model calls, potentially offsetting the savings. To address the challenge, we introduce Differentiated Context Compression (DiffContext), a task-aware, plug-in budget controller for existing compressors with adjustable retention budgets. Keeping the compressor and task model frozen, DiffContext uses offline task feedback to estimate harm and token cost across budgets and construct input-dependent policies. Because predicted harm alone does not guarantee risk control, independent calibration selects a deployment policy that limits the population frequency of utility drops beyond a specified tolerance relative to full-context inference, with a finite-sample guarantee under i.i.d. calibration and deployment sampling. At deployment, the selected policy chooses a budget from input features, requiring at most one compressor call and one task-model call per input. Across twelve configurations spanning three datasets and five compression backbones, DiffContext increases mean task-model token-cost savings from 3.61% for certified fixed budgets to 12.14% on the evaluated budget grid, under the same 10% limit on harm frequency. These results support adaptive budget control as a reusable way to retain compression savings under an explicit task-risk requirement. The code and data are available at https://anonymous.4open.science/r/DiffContext-v1.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Context Compression, Large Language Models, Adaptive Budget Selection, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KeQyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DiffContext selects an input-dependent compression budget for a frozen compressor and task model. It learns harm and cost estimates, then uses independent calibration to control the population harm rate under stated assumptions. Across 12 configuratins, it reports greater token savings than the tested calibrated fixed-budget baselines.

Critical strengths

(1) The framework connects adaptive compression with explicit risk control. Independent calibration and a full-context fallback make the safety criterion clear. (2) Experiments cover several datasets and compressors, with matched calibration comparisons. They show that calibration substantially reduces the harm observed with an uncalibrated controller.

Critical weaknesses

(1) The value of learned input-specific harm estimates is unclear. Simpler calibrated controls are competitive, while the fixed-budget baseline uses only four retention levels. Finer budgets near full context could reduce the reported advantage. (2) The cost analysis includes task-model inference and fit-label generation, but excludes compressor execution and calibration. These costs may materially change the deployment break-even point.

Questions for authors

(1) What advantage remains over simpler calibrated controls and a finer fixed-budget grid under the same calibration protocol? (2) What is the break-even point after including compressor execution and calibration costs?

Reviewer_Uu6HOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DiffContext, a controller that chooses retention budgets for an existing context compressor. The compressor and task model remain frozen. Offline evaluations are used to learn harm and cost estimates, which define a family of input-dependent budget policies. Independent calibration then selects a policy using fixed-sequence testing, providing a finite-sample guarantee on the population frequency of utility drops under the stated i.i.d. assumptions.

Experiments use Qwen-Plus across three datasets and five compression backbones. At a 10% harm limit, DiffContext reports mean task-model token savings of 12.14%, compared with 3.61% for certified fixed budgets.

Critical strengths
  • The risk-control procedure is carefully specified. The paper separates policy construction from independent calibration and freezes the candidate policies and testing order before calibration. The proof in Appendix C appears sound under the stated assumptions. This makes the guarantee meaningful even when the learned harm estimates are inaccurate.

  • The controller offers a useful interface for existing compressors. It chooses the retention budget without changing the compressor or task model, and deployment requires at most one compressor call and one task-model call. The experiments across several compression backbones help demonstrate the usefulness of this design.

Critical weaknesses

The benefit of the full harm predictor remains unclear. The restricted-feature controls in Table S10 are competitive. For example, on HotpotQA–Dense, Length-only achieves 18.12% token savings with 4.44% observed harm, compared with 14.47% savings and 5.56% harm for DiffContext. On HotpotQA–Selective Context, Length-only also has both higher savings and lower observed harm. All these methods use the same certification procedure. The nested fit-split analysis in Table S2 further reports mean per-budget AUROC values of 0.392–0.538 for the stored-label variant. These results leave an important question about the method: when does the richer predictor help budget selection enough to justify its additional complexity?

The experiments do not establish how stable the savings are across fitting and calibration samples. Each configuration uses 90 or 240 fit inputs and 180 calibration inputs. The bootstrap analysis in Appendix D.3 conditions on the fitted and selected policies, so it does not measure variation from refitting the predictor or selecting a different policy during calibration. This matters for the main comparison: certified fixed budgets return full context in eleven configurations, and the calibration threshold is sensitive to a single event—eight harm events pass, while nine fail. The current results therefore leave uncertainty about how consistently the reported savings advantage would recur.

Questions for authors
  • In Table 1, Fixed 0.75 achieves 27.78% CTS and 6.67% test harm on HotpotQA–Dense, yet certified fixed returns full context. What were this budget’s calibration harm count and upper bound?

  • How much of the improvement in Table 1 comes from the learned harm predictor? What evidence shows it adds value beyond a simple length-based policy using the same certification procedure?

  • In Table S10, Length-only achieves both higher savings and lower observed harm than DiffContext on HotpotQA–Dense and HotpotQA–Selective Context. What explains these differences?

Reviewer_MrNeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DiffContext adds input-dependent budget selection to existing context compressors while keeping both the compressor and downstream task model frozen. It uses offline evaluations across retention budgets to predict compression-induced harm and estimate token costs. These estimates define candidate budget policies, which are independently calibrated to select a deployment policy satisfying a population-level harm-rate constraint. The main contribution is the integration of adaptive budget allocation with statistical risk control, rather than a new compression algorithm. Across twelve configurations involving Qwen-Plus, three datasets, and five compressors, mean task-model token savings increase from 3.61% to 12.14% over certified fixed budgets under the same 10% harm-rate limit. Additional experiments examine risk-limit sensitivity, independent calibration, and amortization of offline label-generation costs.

Critical strengths

S1. Clear separation between risk prediction and risk assurance. Rather than treating predicted harm as a reliability guarantee, the method independently evaluates the final budget policies. Under the stated i.i.d. assumptions, this provides a finite-sample population-risk guarantee without requiring monotone risks across budgets. Accommodating prediction errors and non-monotonic compression effects is an important strength of the design. The controlled comparison also supports its practical role: independent calibration reduces mean test harm from 14.03% to 3.98%, albeit with more conservative compression.

S2. Practical modularity supported by multi-configuration evaluation. Budget selection is separated from content compression, requiring no changes to the existing compressor or task model. Deployment avoids repeated downstream evaluations of alternative budgets, requiring at most one compressor call and one task-model call per input. Under the same certification requirement, fixed budgets revert to full context in eleven configurations, whereas DiffContext retains nonzero savings in eleven. This demonstrates the value of input-dependent allocation when a uniform budget is overly conservative and supports the method’s usefulness as an additional budget-selection component.

Critical weaknesses

W1. Positioning relative to closely related work needs clarification. PoC (Zhao et al., 2026; arXiv:2603.19733) also uses predicted downstream performance to select input-dependent compression ratios for an existing compressor, yet it is not discussed in the manuscript. Independent risk certification is a meaningful distinction of DiffContext. However, the differences in prediction targets, budget-selection rules, and quality constraints should be made explicit. A comparison under matched conditions, potentially including a prediction-based baseline equipped with the same certification procedure, would help establish the additional contribution of DiffContext’s policy design.

W2. Unclear justification and added value of the harm predictor. Appendix B.2 considers three classifiers but does not report which model is selected for each configuration. Moreover, Table S10 shows no consistent advantage of the full predictor over simpler Budget-only and Length-only variants in terms of token savings and task performance under the same risk certification procedure. Could the authors clarify the classifier choices and discuss whether the current statistical features adequately capture task-relevant evidence, compared with richer semantic representations? These results leave the contribution of the core prediction module insufficiently established.

Questions for authors

Please see Weaknesses.

1 comment

Thanks for the response. I will keep my current positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.