- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CLESO learns to organize retrieved text so language models can select supporting evidence for specialized questions within a limited reading budget.
Critical strengths
-
The experiments are extensive, covering three domains, multiple baselines, different observation budgets, and component analyses that help assess the method.
-
The paper is well written, and the figures are carefully designed, providing clear presentations of the motivation and experimental results.
Critical weaknesses
-
Evidence F1 gains over the strongest baselines are below two percentage points, without paired confidence intervals establishing that the improvements are reliable.
-
The ablations support budget allocation and evidence grouping, but do not establish whether learning the interval structure is necessary for the gains.
Questions for authors
-
Could you report paired confidence intervals for the main improvements and results across training seeds to assess their reliability?
-
How do learned intervals compare with ordinary partitions when the budget allocator, evidence selection procedure, and group sizes are held fixed?
4 comments
For questions
Q1: Could you report paired confidence intervals for the main improvements and results across training seeds to assess their reliability?
A1: Table 2 reports mean ± sample SD across seeds 13, 17, and 29. Separately, we compute paired CIs within each seed using bootstrap resamples, clustering by paper for QASPER and by question for BioASQ and LegalBench-RAG.
Against the fixed strongest mean-F1 comparator in each benchmark, CP–comparator differences and pointwise 95% CIs are:
| Benchmark / comparator | Seed 13 | Seed 17 | Seed 29 |
|---|---|---|---|
| BioASQ / RankGPT | +2.48 [−0.78, 6.16] | +1.91 [−1.20, 5.35] | +1.25 [−1.66, 4.55] |
| QASPER / SetR | −0.17 [−2.21, 1.84] | −0.26 [−2.05, 1.58] | +1.02 [−0.79, 2.82] |
| LegalBench-RAG / Qwen3-Reranker-4B | +0.37 [−2.15, 2.92] | +0.61 [−1.61, 2.77] | +0.64 [−1.60, 2.88] |
CP exceeds these comparators in 3/3, 1/3, and 3/3 runs, respectively, but all corresponding intervals include zero.
The bootstrap quantifies case-sampling uncertainty conditional on each evaluated run; sample SD describes observed cross-seed variation.
Q2: How do learned intervals compare with ordinary partitions when the budget allocator, evidence selection procedure, and group sizes are held fixed?
A2: We compare learned organization against contiguous, approximately equal-sized RAW partitions using each question's PAPER-predicted interval count. With the same budget allocator and downstream Probe–CP–Selector configuration, PAPER improves mean evidence F1 by 0.13/2.84/5.54 points on BioASQ/QASPER/LegalBench-RAG. The larger differences on QASPER and LegalBench-RAG support learned organization beyond partition count alone. The BioASQ difference is small. Our separate package-size-controlled comparisons address that confound. We have updated the main-text ablation study accordingly, with the additions highlighted in blue.
Thank you for the response. I will maintain my original positive rating.
Dear Reviewer BdcB,
Thank you for reviewing our responses and for your continued positive assessment. We appreciate your constructive suggestions on the structural controls and statistical reporting, which helped improve the revision. We remain available for any further questions.
Best regards,
The authors
Thank you for noting the breadth of the evaluation and the clarity of the presentation. We focus below on the reliability of the reported gains and the contribution of learned structure relative to simpler alternatives. W1 and Q1 refer to Weakness 1 and Question 1, respectively, with analogous labels for the remaining points. Related experiments and information were updated on lated draft documents.
W1: Evidence F1 gains over the strongest baselines are below two percentage points, without paired confidence intervals establishing that the improvements are reliable.
R1: We selected BioASQ, QASPER, and LegalBench-RAG to evaluate performance across both domains and substantially different processing scales. Their candidate sequences average approximately 885k, 4.9k, and 2.08M tokens, respectively. Among target-positive questions, annotated evidence counts range from 1–141, 1–133, and 1–31, averaging 35.69, 9.49, and 2.64 per question, respectively. These differences in candidate-pool size and evidence requirements challenge methods to maintain evidence coverage while controlling processing costs. Accordingly, improvements should be interpreted within each benchmark's evaluation setting rather than compared directly across datasets. To assess reliability, we added three-seed results and paired confidence intervals, separating mean differences from statistical evidence. Against the strongest baseline by three-seed mean evidence F1, the updated results are:
F1 values are mean ± sample SD in percent; differences are percentage points. All nine per-seed paired CIs for these comparisons include zero, so the higher means do not establish superiority over the strongest baselines. Q1 below provides the intervals.
Stronger evidence appears in the CP–IP comparison: gains of 14.20–14.60 points on QASPER and 4.90–5.85 points on LegalBench-RAG have positive pointwise CIs in all three seeds; BioASQ excludes zero only in seed 17. We therefore distinguish the overall packing-policy benefit from the more modest strongest-baseline differences. These statistical results are included in the revised main table and appendix.
W2: The ablations support budget allocation and evidence grouping, but do not establish whether learning the interval structure is necessary for the gains.
R2: Our discussion on the budget and grouping focused on the core objective of cost control. In order to discuss the core logic of the revenue, we conducted a controlled sequence ablation while keeping the downstream Probe, proposal generation, CP packing, Selector, and budget rules unchanged. Probe and observation each retain an independent 3,250-token cap. We reuse checkpoints trained on the training splits, without Test fine-tuning. The four conditions and their final Evidence F1 (%) are:
PAPER exceeds HEAD by 0.40, 1.02, and 2.68 percentage points, and RAW-Uniform by 0.13, 2.84, and 5.54 points, respectively. These comparisons support an additional benefit from criterion-guided sequence learning over ordinary global-head optimization and simple partitioning under the same downstream pipeline, with the largest gains on LegalBench-RAG. We have added this controlled ablation to Section 6.4 (Table 6), with revisions highlighted in blue.