- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work argues that longer contexts do not necessarily improve LLM performance on RAG tasks and proposes CALIBER for training-free context management. Specifically, during offline calibration, the authors construct EQ as the product of answer correctness (Pans), contextual support (Fsup), and consistency across multiple sampled responses (Cresp). They then estimate an effective context boundary, τeff, using sustained quality degradation and a consistency threshold. During online inference, this boundary guides the organization of hierarchical indices and subcontexts. Vector retrieval and LLM-based routing are followed by parallel generation and recursive aggregation into a final answer.
Critical strengths
S1 The research problem is relevant to practical deployment. The paper examines the gap between a model’s advertised context window and the amount of context it can reliably use, and attempts to translate quality measurements into an actionable context budget. This problem is directly relevant to configuring systems for long-document question answering and retrieval-augmented generation.
S2 The method requires no fine-tuning and offers a relatively complete system design. CALIBER uses existing models for offline calibration and applies budget constraints throughout source-subcontext organization, parallel inference, and recursive aggregation, without updating the target model’s weights. The framework offers a practical approach to implementing context management with existing models.
Critical weaknesses
W1 The relevance of the evaluated models and the need for the method on stronger models require further validation. My main concern is that the observed degradation with increasing context length may be confined to weaker models. In particular, the main experimental results cover only Gemini 2.5 Pro, GPT-4o, and Claude Sonnet 4.5. The existing length-bucket curves therefore do not establish that longer contexts generally weaken the capabilities of stronger models.
W2 The first stage depends heavily on the calibration tasks, with insufficient evidence of generalization and broad applicability. τeff explicitly depends on the model, calibration workload, and evaluation protocol, while correctness evaluation during calibration requires reference answers.
W3 Systematic comparisons with other context management methods are lacking. The main baselines in Table 2 are limited to Dense RAG and Self-RAG. RULER primarily provides an alternative budget in Table 3 and does not cover different context management strategies. The authors have not sufficiently demonstrated advantages over approaches based on grouped long-context reading, context compression, hierarchical summarization, or adaptive budgeting.
Questions for authors
Q1 On more recent and stronger long-context models, does consistent quality degradation still occur when only context length is varied while the question and relevant evidence are held fixed? Does CALIBER still offer clear benefits over direct long-context inference on these models?
Q2 Could the authors calibrate τeff only on source tasks, keep it fixed, and test its transfer to unseen tasks or domains? Could they also compare quality and cost against other context management strategies using the same document corpus and comparable computational budgets?
5 comments
Quality and computational trade-offs. CALIBER achieves the highest score (49.2), outperforming grouped reading (LLM×MapReduce, +6.4 points), context compression (LLMLingua-2 22K, +3.3), and hierarchical retrieval (MiA-RAG top-10, +2.2). Despite its multi-stage architecture, CALIBER processes only 150.3K total input tokens, compared with 372.6K, 346.8K, and 860.8K for these alternatives, corresponding to reductions of 59.7%, 56.7%, and 82.5%, respectively.
These results suggest that CALIBER's targeted routing and bounded subcontext processing can retain relevant distributed evidence more effectively than the evaluated compression and grouped-reading approaches, while avoiding repeated processing of large portions of the source corpus. Unlike approaches that rely primarily on compressed representations or broad batch processing, CALIBER generates partial answers directly from selected raw evidence and aggregates them within calibrated context budgets.
Importantly, the reported total input includes upstream processing as well as generation, revealing costs that generation-only comparisons may overlook. While these results demonstrate favorable quality–token trade-offs in the evaluated setting, token counts do not directly measure FLOPs, latency, or monetary cost.
Dear Reviewer E8TC,
We know discussion periods get busy for everyone, and we appreciate the time you've already put into this review. With the window now closing, we wanted to follow up: we posted a response to W1–W3 and Q1–Q2, including the new controlled-length experiment isolating degradation from over-window coverage, the τ_eff transfer and stability results on unseen data, and the extended comparison against Adaptive-k, compression-based, and batch-reading baselines.
We'd really value hearing whether this addresses your concerns, particularly around model strength and systematic baseline coverage, or whether there's anything you'd like us to dig into further. We're glad to keep discussing for as long as is useful before the deadline.
Thank you for pushing us to tighten these comparisons — it's made the evaluation substantially stronger.
[W1, Q1] Is long-context degradation still a problem?
Thank you for requesting this controlled evaluation. Long-context limitations remain relevant even for strong, recent models, although their severity varies across models and input conditions. We evaluated 120 items from SQuAD, HotpotQA, and FanOutQA, keeping questions and relevant evidence fixed while extending contexts to 8–128K with distractor passages, unrelated text, or neutral filler. Degradation was predefined as a loss of at least 5 points with a paired-bootstrap 95% confidence interval excluding zero.
Even among recent API models, Haiku 4.5 met this criterion in 2/6 tested experimental conditions and GPT-5.2 in 1/6, with maximum losses of 16.9 and 5.7 points. The other seven API models did not meet the criterion. Degradation was more frequent for Qwen3-4B (8/9 configurations, maximum loss of 27.5 points) and Llama-3.1-8B (5/9, 18.1 points). These findings show that stronger models can still exhibit length sensitivity and motivate calibrating the operating budget to each model.
Table R.4. CALIBER vs. Truncated Direct on Over-Window Documents (FanOutQA)
Benefits beyond the available window. Models that remain robust within their window still face documents exceeding their single-call capacity. On such held-out FanOutQA documents, CALIBER outperforms truncated Direct by 8.01 points [95% CI: 5.29, 10.82]. This demonstrates the benefit of bounded processing and aggregation for over-window inputs. Together, these results support CALIBER’s relevance to strong models through both model-dependent context calibration and processing of documents beyond a single-call window.
[W2, Q2] What does τeff predict, and does it transfer?
Thank you for raising this question. CALIBER estimates a model- and workload-dependent operating budget from observed response quality. We examine whether this budget captures behavior beyond its calibration data and remains useful without target-specific tuning.
Predictive relevance and transfer. On an independent controlled-length evaluation, models with longer calibrated boundaries exhibit smaller performance declines from 8K to 32K, with a strong rank association across five models (Spearman ρ=0.90). This suggests that τeff captures meaningful differences in context sensitivity. On unseen LongBench v2, applying the fixed boundary without retuning yields positive mean gains over a 128K budget for both Haiku (+2.11 points, 95% CI [−3.52, +8.45]) and GPT-5.2 (+2.21 points, [−4.41, +8.82]). These results provide encouraging evidence for transfer, although the confidence intervals do not establish significant improvements.
Calibration stability and label dependence. Leaving out one calibration source shifts the boundary by at most one bucket in 37/55 cases (67%), indicating stability across source compositions. The reference-free Fsup×Cresp variant also reproduces the full Pans×Fsup×Cresp boundary in 9/13 models, suggesting feasibility without reference answers. These findings support reusing calibrated budgets across related workloads with reduced dependence on labeled data, while recommending recalibration for substantial task or domain shifts.
[W3, Q2] Comparison to alternative context-management strategies
We evaluated CALIBER against adaptive retrieval, grouped reading, context compression, and hierarchical retrieval methods on matched items and models.
Table R.5. CALIBER improvements (points, 95% CI).
All five comparisons show statistically significant gains, including adaptive retrieval with Adaptive-k.
Table R.6. Additional strategy comparison (Qwen2.5-14B-Instruct).
Tables R.5–R.6 show CALIBER's quality gains across adaptive retrieval, grouped reading, compression, and hierarchical retrieval, with lower total input-token usage in Table R.6.