acceptodds
Under review as a conference paper at ICLR 2027

CLESO: Criterion-guided Laminar Evidence Sequence Organization for Specific Domain Budgeted Probative Evidence Selection

Abstract

Evidence selection in domain-specific question answering aims to identify a compact set of text units from a large-scale corpus that jointly supports an answer. Building on high-recall candidate sequences produced by relevance ranking, existing systems improve evidence selection through instructions, iterative reasoning, or set-level objectives. However, evidence set selection in domain-specific QA must locate support that may be dispersed across long candidate sequences and assess whether a candidate set jointly enables a reliable answer without mixing incompatible entity conditions or scopes. Repeatedly assessing alternative sets with an LLM is costly. We propose CLESO, a structural sequence distillation framework that learns laminar candidate sequences to prioritize access to potential support under a limited observation budget before downstream joint selection. During supervision, ontology-grounded criteria provide signal for candidate qualifications and cross-evidence dependencies, and a Laminar compiler turns these signals into local ordering, head-admission, and boundary-separation targets. At inference, a lightweight Transformer refiner predicts residual scores from inference-available features, and a fixed decoder recovers intervals from the resulting sequence. The recovered intervals guide budgeted local observation before LLM-based package selection, allowing the learned structure to guide evidence access without executing criterion search at inference On BioASQ, QASPER, and LegalBench-RAG, CLESO-CP gains an average of 1.45 evidence-F1 points over the strongest baseline, with a 3.25k-token observation cap on queues averaging 1M tokens.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Budgeted evidence selection, Structural sequence distillation, Laminar intervals, Domain-specific question answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BdcBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CLESO learns to organize retrieved text so language models can select supporting evidence for specialized questions within a limited reading budget.

Critical strengths
  1. The experiments are extensive, covering three domains, multiple baselines, different observation budgets, and component analyses that help assess the method.

  2. The paper is well written, and the figures are carefully designed, providing clear presentations of the motivation and experimental results.

Critical weaknesses
  1. Evidence F1 gains over the strongest baselines are below two percentage points, without paired confidence intervals establishing that the improvements are reliable.

  2. The ablations support budget allocation and evidence grouping, but do not establish whether learning the interval structure is necessary for the gains.

Questions for authors
  1. Could you report paired confidence intervals for the main improvements and results across training seeds to assess their reliability?

  2. How do learned intervals compare with ordinary partitions when the budget allocator, evidence selection procedure, and group sizes are held fixed?

4 comments

Thank you for noting the breadth of the evaluation and the clarity of the presentation. We focus below on the reliability of the reported gains and the contribution of learned structure relative to simpler alternatives. W1 and Q1 refer to Weakness 1 and Question 1, respectively, with analogous labels for the remaining points. Related experiments and information were updated on lated draft documents.

W1: Evidence F1 gains over the strongest baselines are below two percentage points, without paired confidence intervals establishing that the improvements are reliable.

R1: We selected BioASQ, QASPER, and LegalBench-RAG to evaluate performance across both domains and substantially different processing scales. Their candidate sequences average approximately 885k, 4.9k, and 2.08M tokens, respectively. Among target-positive questions, annotated evidence counts range from 1–141, 1–133, and 1–31, averaging 35.69, 9.49, and 2.64 per question, respectively. These differences in candidate-pool size and evidence requirements challenge methods to maintain evidence coverage while controlling processing costs. Accordingly, improvements should be interpreted within each benchmark's evaluation setting rather than compared directly across datasets. To assess reliability, we added three-seed results and paired confidence intervals, separating mean differences from statistical evidence. Against the strongest baseline by three-seed mean evidence F1, the updated results are:

BenchmarkComparatorComparator F1CLESO-CP F1Mean difference
BioASQRankGPT22.68 ± 0.4324.56 ± 0.20+1.88
QASPERSetR57.43 ± 0.4857.63 ± 0.29+0.20
LegalBench-RAGQwen3-Reranker-4B22.75 ± 0.1123.29 ± 0.19+0.54

F1 values are mean ± sample SD in percent; differences are percentage points. All nine per-seed paired CIs for these comparisons include zero, so the higher means do not establish superiority over the strongest baselines. Q1 below provides the intervals.

Stronger evidence appears in the CP–IP comparison: gains of 14.20–14.60 points on QASPER and 4.90–5.85 points on LegalBench-RAG have positive pointwise CIs in all three seeds; BioASQ excludes zero only in seed 17. We therefore distinguish the overall packing-policy benefit from the more modest strongest-baseline differences. These statistical results are included in the revised main table and appendix.

W2: The ablations support budget allocation and evidence grouping, but do not establish whether learning the interval structure is necessary for the gains.

R2: Our discussion on the budget and grouping focused on the core objective of cost control. In order to discuss the core logic of the revenue, we conducted a controlled sequence ablation while keeping the downstream Probe, proposal generation, CP packing, Selector, and budget rules unchanged. Probe and observation each retain an independent 3,250-token cap. We reuse checkpoints trained on the training splits, without Test fine-tuning. The four conditions and their final Evidence F1 (%) are:

ConditionSequence and partitionBioASQQASPERLegalBench-RAG
RAWOriginal Qwen Embedding scores and ordering23.7852.9521.20
HEADGlobal-head Refiner with the same architecture and input features as PAPER25.8254.4026.61
RAW-UniformRAW ordering with equal-sized contiguous partitions, using CLESO's predicted interval count KK26.0952.5823.75
CLESOSubmitted criterion-supervised Teacher–Student sequence26.2255.4229.29

PAPER exceeds HEAD by 0.40, 1.02, and 2.68 percentage points, and RAW-Uniform by 0.13, 2.84, and 5.54 points, respectively. These comparisons support an additional benefit from criterion-guided sequence learning over ordinary global-head optimization and simple partitioning under the same downstream pipeline, with the largest gains on LegalBench-RAG. We have added this controlled ablation to Section 6.4 (Table 6), with revisions highlighted in blue.

For questions

Q1: Could you report paired confidence intervals for the main improvements and results across training seeds to assess their reliability?

A1: Table 2 reports mean ± sample SD across seeds 13, 17, and 29. Separately, we compute paired CIs within each seed using bootstrap resamples, clustering by paper for QASPER and by question for BioASQ and LegalBench-RAG.

Against the fixed strongest mean-F1 comparator in each benchmark, CP–comparator differences and pointwise 95% CIs are:

Benchmark / comparatorSeed 13Seed 17Seed 29
BioASQ / RankGPT+2.48 [−0.78, 6.16]+1.91 [−1.20, 5.35]+1.25 [−1.66, 4.55]
QASPER / SetR−0.17 [−2.21, 1.84]−0.26 [−2.05, 1.58]+1.02 [−0.79, 2.82]
LegalBench-RAG / Qwen3-Reranker-4B+0.37 [−2.15, 2.92]+0.61 [−1.61, 2.77]+0.64 [−1.60, 2.88]

CP exceeds these comparators in 3/3, 1/3, and 3/3 runs, respectively, but all corresponding intervals include zero.

The bootstrap quantifies case-sampling uncertainty conditional on each evaluated run; sample SD describes observed cross-seed variation.

Q2: How do learned intervals compare with ordinary partitions when the budget allocator, evidence selection procedure, and group sizes are held fixed?

A2: We compare learned organization against contiguous, approximately equal-sized RAW partitions using each question's PAPER-predicted interval count. With the same budget allocator and downstream Probe–CP–Selector configuration, PAPER improves mean evidence F1 by 0.13/2.84/5.54 points on BioASQ/QASPER/LegalBench-RAG. The larger differences on QASPER and LegalBench-RAG support learned organization beyond partition count alone. The BioASQ difference is small. Our separate package-size-controlled comparisons address that confound. We have updated the main-text ablation study accordingly, with the additions highlighted in blue.

Thank you for the response. I will maintain my original positive rating.

Dear Reviewer BdcB,

Thank you for reviewing our responses and for your continued positive assessment. We appreciate your constructive suggestions on the structural controls and statistical reporting, which helped improve the revision. We remain available for any further questions.

Best regards,
The authors

Reviewer_ER64OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CLESO, a framework for selecting evidence from long relevance-ranked candidate sequences under a limited observation budget. During training, ontology-grounded criteria and annotated evidence induce interval partitions, local ordering priorities, and score-separation targets. A lightweight Transformer learns residual scores, from which a fixed decoder recovers intervals at inference. A separately trained Probe allocates observation budgets across interval heads. Proposal-conditioned packing and an LLM selector then construct the final evidence set. The paper also derives conditional guarantees for interval recovery and supporting-witness exposure. Experiments on BioASQ, QASPER, and LegalBench-RAG report evidence-F1 improvements reported baselines. Ablation studies are also set to prove the necessity of components.

Critical strengths
  1. The paper addresses a relevant and practically important problem in evidence-grounded QA. The motivation for finding complementary evidence under a limited observation budget is clear.

  2. The framework combines ontology-guided supervision with learned evidence access in an interesting way. Keeping criterion search offline is a reasonable design choice for reducing inference-time complexity.

  3. Comprehensive experiments are done to evaluate the model from different perspectives. Also the budget sensitivity analysis and component ablations evaluate the framework from multiple perspectives. These results provide useful insights into how observation allocation and evidence packing affect performance.

Critical weaknesses
  1. The contribution of criterion-guided structural learning is not sufficiently isolated. Comparisons with simpler partitions or directly supervised ranking are needed to establish the value of this central component.

  2. The packing comparison changes both cross-interval composition and package granularity. It remains unclear which factor drives the substantial precision improvements.

  3. The F1 gains over the strongest baselines are relatively modest, and their statistical reliability is unclear without paired uncertainty estimates.

  4. The efficiency comparison excludes feature-construction costs and uses different accounting boundaries across methods. This limits the conclusions that can be drawn about end-to-end computational savings.

Questions for authors
  1. How does the full Probe+CP pipeline perform when criterion-guided structural supervision is replaced by simple interval partitions or a directly supervised scoring model using the same available features and evidence labels?

  2. What are the stage-level costs under the same accounting boundary for CLESO and the strongest practical baselines, including question-dependent feature construction and downstream selection?

  3. How many questions contribute to each metric for each method, and which exclusions are method-dependent? Please clarify the Answer/PB judge configurations and reference qualification procedure, and provide paired differences with confidence intervals on common eligible cases.

4 comments

Thank you for identifying the need to isolate structural contributions and make the evaluation auditable. The revision adds controlled sequence and packing experiments, paired uncertainty, and expanded cost accounting. We distinguish these completed additions from the remaining denominator, Judge-comparability, and end-to-end timing limitations.

W1: Contribution of criterion-guided structural learning

R1: In CLESO, the contribution of structure learning is mainly achieved through low-cost computational logic to ensure equal exposure of different depths of evidence and to support the consideration of combinations of evidence at different depths.Table 6 now directly compares criterion-guided refinement with ordinary global-head training and simple contiguous partitioning. Specifically, HEAD shares CLESO's architecture and input features; RAW-Uniform preserves RAW ordering and matches CLESO's predicted interval count. All conditions share frozen downstream models and the Probe–proposal–CP–Selector procedure, with independent 3,250-token Probe and observation budgets.

BenchmarkRAWHEADRAW-UniformCLESO
BioASQ23.7825.8226.0926.22
QASPER52.9554.4052.5855.42
LegalBench-RAG21.2026.6123.7529.29

Final evidence F1 (%) improves over HEAD by 0.40/1.02/2.68 points, and over RAW-Uniform by 0.13/2.84/5.54 points. These results support learned organization beyond ordinary head optimization and partition count within this pipeline. In revision draft, Table 3 separately reports historical-cohort, single-seed evidence-label fine-tuning. Fine-tuned BGE leads CLESO-CP on QASPER and LegalBench-RAG (59.10/37.34 versus 56.25/29.98), whereas CLESO-CP leads on BioASQ. Thus, the controlled structural contribution and overall supervised competitiveness are distinct findings. Both comparisons have been added to the main text with blue text.

W2: Packing composition versus granularity

R2: Table 7 adds CP/IP comparisons with the same observation pool, proposal, scorer, and Selector under common package-size settings M=2,3,4,5M=2,3,4,5.

Representative comparisonCP−IP F1Pointwise 95% CI
LegalBench-RAG, M=2M=2+1.67[0.21, 3.24]
QASPER, M=2M=2+0.23[−0.03, 0.51]
BioASQ, M=5M=5−2.42[−3.68, −1.20]

All QASPER intervals include zero. The mixed results show that composition and granularity jointly affect selection; cross-interval packing is not uniformly advantageous at matched settings. We therefore interpret the original CP–IP ablation as evidence for the overall packing policy.

W3: Magnitude and reliability of the main F1 gains

R3: We chose BioASQ, QASPER, and LegalBench-RAG, taking into account not only cross-domain evaluations but also the significant differences in their processing scales. The average candidate queue lengths of the three benchmarks are approximately 885k, 4.9k, and 2.08M tokens, respectively. In the questions containing labeled positive examples, the number of target evidences ranges from 1–141, 1–133 and 1–31 respectively, with an average of 35.69, 9.49 and 2.64 evidences per question. These benchmarks cover different scenarios ranging from short document sequences to extremely long candidate pools, and the differences in the number of target evidences are significant, making them suitable for testing the ability of the methods to balance evidence coverage, combination selection, and processing costs under a limited budget.

In revision darft, Table 2 now reports three seeded runs. CP achieves evidence F1 of 24.56±0.20, 57.63±0.29, and 23.29±0.19% on BioASQ, QASPER, and LegalBench-RAG. Mean differences against the strongest mean-F1 baselines are +1.88/+0.20/+0.54 points.

Benchmark/comparatorSeed 13: Δ [95% CI]Seed 17: Δ [95% CI]Seed 29: Δ [95% CI]
BioASQ / RankGPT+2.48 [−0.78, 6.16]+1.91 [−1.20, 5.35]+1.25 [−1.66, 4.55]
QASPER / SetR−0.17 [−2.21, 1.84]−0.26 [−2.05, 1.58]+1.02 [−0.79, 2.82]
LegalBench-RAG / Qwen3-Reranker+0.37 [−2.15, 2.92]+0.61 [−1.61, 2.77]+0.64 [−1.60, 2.88]

These are per-seed pointwise intervals from paired bootstrap resamples, clustered by paper for QASPER and by question elsewhere.

The CP–IP comparison provides clearer repeated evidence: gains of 14.20–14.60 points on QASPER and 4.90–5.85 on LegalBench-RAG have positive intervals in every seed.

W4: Feature-construction costs and accounting boundaries

R4: The supplementary accounting includes question-dependent feature processing, query encoding, refinement, Probe, package scoring, and generation/selection. Corpus-side ontology annotations are cached; criterion search is performed during training.

We separately report a model-workload proxy, ∑mPmLm/1012\sum_m P_m L_m / 10^{12}, and measured CPU component time. Processing length counts tokens for text models and candidate positions for the Refiner, making the proxy a custom workload measure rather than FLOPs or GPU runtime.

Cost comparisonBioASQLegalBench-RAG
BGE model workload / CP2.84×12.98×
Qwen3-Reranker model workload / CP42.82×143.65×
CP measured CPU components, s/question0.7100.475
BGE measured CPU components, s/question8.67129.918

These measurements support lightweight full-queue refinement followed by expensive scoring restricted to observed evidence. The benefit is conditional: on QASPER, CP requires 7.10× BGE's and 1.94× Qwen3's workload, and exceeds the evaluated SetR configuration on all three benchmarks.

Figure 4 and Tables 14–15 provide the expanded accounting. They support a lower workload relative to full-queue reranking on long candidate pools.

For questions:

Q1: Replacing structural supervision with simpler alternatives

A1: HEAD supplies the directly supervised scoring control with the same architecture and available features; RAW-Uniform supplies the simple partition control. Under the unchanged downstream procedure, PAPER exceeds RAW by 2.44/2.47/8.09 points and HEAD by 0.40/1.02/2.68 points on BioASQ/QASPER/LegalBench-RAG.

This tests the total contribution of sequence organization within the pipeline. Actual observations and packages can change with the sequence, so it does not isolate interval boundaries independently of every downstream consequence. Table 6 and its discussion are included in Section 6.4 and highlighted in blue.

Q2: Stage-level costs under a common boundary

A2: Tables 14–15 report stage-level workload and measured CPU components separately for CLESO and the baselines. For CP, the principal recorded model stages are:

CP model-workload proxyBioASQLegalBench-RAG
Package scoring265.960115.971
Probe11.63613.341
Generation/selection45.68141.804
Total, including encoding and refinement323.316171.162

These per-question values use the proxy defined in W4; totals retain unrounded aggregates. CPU totals are 0.710/0.475 seconds per question, reported separately. This breakdown identifies repeated package scoring as the dominant model-workload component and includes question-dependent preprocessing.

Q3: Metric eligibility, Judge configuration, and paired differences

A3: Answer evaluates generated answers; PB evaluates whether selected evidence supports all frozen requirements. QASPER Answer uses official best-reference token F1 without an Answer Judge. BioASQ/LegalBench-RAG Answer and all PB scores are reference-conditioned, model-assisted binary assessments. Evidence F1 does not use these Judges.

For audited CP/IP requests, both Judges use Qwen3.6-35B-A3B, BF16, temperature 0, thinking disabled, and strict JSON Schema, with separate prompts. Each outcome has one valid judgment and at most two technical attempts.

BenchmarkReported evidence-F1 NAnswer valid NPB valid N
BioASQ260244240
QASPER1,4511,451835
LegalBench-RAG1,412844815

References were checked for scope, necessity, source support, and completeness before inspecting method outputs. The available summaries report all 18 CP–IP Answer/PB intervals as including zero. For illustration, seed-13 CP–SetR differences are +9.09 [5.19, 13.33] for QASPER PB and +10.43 [5.69, 15.17] for LegalBench-RAG Answer

Dear Reviewer ER64,

Thank you for asking what learned intervals contribute before LLM selection and at what cost. First, three benchmarks illustrate the challenge: the final evidence budget is fixed at 3,250 tokens, but candidate pools average 4.9k tokens on QASPER, 885k on BioASQ, and 2.08M on LegalBench-RAG. The longest pools also have the fewest targets: among questions with annotated evidence, LegalBench-RAG averages 2.64 target passages versus 35.69 on BioASQ. This shows the diversity of our experiment.

As for our task, passages that support an answer together may lie far apart in a ranked list. CLESO keeps the semantic encoder frozen and uses ontology-grounded criteria to learn local score adjustments. It decodes the refined scores into intervals, probes their prefixes at different depths, and packs observed passages from different intervals so the LLM can assess them together within its evidence budget. The core advantage of CLESO is that the evidence on different sequence depths has equal chance to expose to the LLM for selection, even if the sequence is extremely long and cause cost problem.

Specifically, Table 6 holds the downstream pipeline and budgets fixed. Compared with a global ranking head using the same architecture and features, CLESO’s evidence F1 was 0.40, 1.02, and 2.68 points higher on BioASQ, QASPER, and LegalBench-RAG, respectively. On LegalBench-RAG, it also gained 5.54 points over uniform partitions matched in number.

Furthermore, Figure 4 shows the workload trade-off. Full-pool BGE reranking used 2.84× CLESO-CP’s model-workload proxy on BioASQ and 12.98× on LegalBench-RAG, but less on QASPER.

We hope this clarifies CLESO’s central contribution: learning where to look in a ranked list so that passages far apart can reach the same LLM decision within a fixed evidence budget.

We remain available for any further questions you may have.

Best regards,

The Authors

Reviewer_QUFBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CLESO, a framework for evidence selection under constrained observation budgets in domain-specific question answering. During training, an ontology-guided criterion search uses annotated evidence to construct local ordering and interval-boundary supervision for a lightweight Transformer-based refiner. During inference, refined scores recover intervals, a trained Probe allocates reading budgets across its heads, and proposal-conditioned packing followed by LLM selection determines the final evidence set. The paper further provides conditional guarantees for interval recovery and evidence exposure. Experiments on BioASQ, QASPER, and LegalBench-RAG report an average evidence-F1 improvement of 1.45 percentage points over the strongest baselines with a 3,250-token observation pool, supported by budget sweeps, cost analyses, and component ablation studies.

Critical strengths
  1. The paper studies exposing dispersed supporting evidence before expensive joint selection. Compiling criterion-derived priorities and interval boundaries into a lightweight refiner provides a concrete mechanism to learn this access policy without repeating criterion search at inference. This is a meaningful methodological direction beyond independently scoring passage relevance.

  2. CLESO-CP attains the highest reported evidence F1 scores across all three benchmarks, surpassing the strongest baseline by 0.55, 1.93, and 1.86 percentage points, respectively. Budget sweeps and component ablation studies further demonstrate the utility of the complete approach, with allocation enhancing evidence exposure and CP packing contributing to improved final precision.

Critical weaknesses
  1. The ablation studies primarily demonstrate the significance of Probe allocation and CP packing. However, they do not provide evidence that criterion-derived supervision and learned interval structure outperform simpler alternatives when the downstream pipeline remains unchanged. Additionally, the CP–IP comparison alters package composition and potentially package granularity. As a result, the observed precision gains do not directly establish the advantage of integrating complementary evidence across intervals. Consequently, it remains uncertain whether the paper’s distinctive structural mechanisms account for the reported improvements.

  2. CLESO leverages domain-specific supervision, whereas several key competitors employ prompt-only or reduced configurations. The shared final-evidence cap does not standardize total processing cost, as CLESO utilizes a distinct probing budget and additional processing stages. Appendix E.5 notes that feature-construction costs are omitted and that baseline stages have not been evaluated within a consistent accounting framework. These limitations are significant because efficiency is a primary motivation, and the improvements over the strongest baselines are relatively modest.

Questions for authors
  1. With the same downstream allocation, CP packing, selector, and observation budgets, how does performance change when you replace the learned structure with fixed-size or original-score-based partitions? Comparing with a conventional evidence-label ranker would also help determine the value of criterion-derived supervision.

  2. Which feature-construction operations can be cached across questions and which must be performed for each new question? Are runtime measurements available for CLESO and the strongest baselines under a common boundary including all question-dependent processing?

  3. Which baselines receive comparable domain supervision? It could be better to provide effective evaluation counts and paired uncertainty estimates for the main evidence-F1 differences to assess the reliability of the reported gains.

6 comments

Thank you for requesting controls that separate structural organization from supervision, packing, and processing costs. The revision adds direct comparisons for these factors and distinguishes supported component-level benefits from the more modest differences against the strongest baselines. W and Q retain the original weakness and question numbering, and related experiment results were updated in lated draft with blue text.

W1: Structural learning and packing granularity

R1: We added two complementary controls.

First, The sequence structure proposed by CLESO based on the prediction of Laminar characteristics is first consistent with the ontology-based criterion. This enables evidence at different depths to have an equal opportunity to be exposed to the selection process under budget constraints. This goal itself is an advantage. To verify this advantage, Table 6 directly tests ordinary ranking and simple partitioning alternatives. HEAD shares PAPER's architecture and input features; RAW-Uniform preserves RAW ordering and matches PAPER's predicted interval count. All conditions use the same frozen downstream models and Probe–proposal–CP–Selector procedure, with independent 3,250-token Probe and observation caps.

BenchmarkRAWHEADRAW-UniformPAPER
BioASQ23.7825.8226.0926.22
QASPER52.9554.4052.5855.42
LegalBench-RAG21.2026.6123.7529.29

Values are final evidence F1 (%). PAPER exceeds HEAD by 0.40/1.02/2.68 points and RAW-Uniform by 0.13/2.84/5.54 points, respectively. These controls support an incremental contribution from learned sequence organization within the shared pipeline, particularly on LegalBench-RAG.

Second, The interval head access logic implemented by CLESO supports the implementation of the CP packing strategy. The absence of a defined packing granularity in this case is due to the fact that in the benchmark, the target quantity is not fixed. Similar to the cases in BioASQ, the target quantity ranges from 2 to 100, so the packing strategy is not suitable for a fixed granularity. Of course, we also conducted packing experiments with strict control of the granularity. Table 7 compares CP/IP under common package-size settings M=2,3,4,5M=2,3,4,5, holding the observation pool, proposal, and scoring/selection procedure fixed. CP improves LegalBench-RAG at M=2M=2: +1.67 [0.21, 3.24] F1 points. All QASPER intervals include zero, while BioASQ favors IP at M=5M=5: −2.42 [−3.68, −1.20]. Thus, the original comparison supports the overall packing policy, not an exclusive attribution to cross-interval complementarity. Both analyses are included in Section 6.4, with revisions highlighted in blue.

W2: Supervision fairness, processing cost, and modest gains

R2: The supervision we implemented in CLESO is not for the entire ranking process, but rather for the structural supervision with relatively smaller perturbations in relevance. In order to compare this kind of supervision with the direct supervision of sequence targets, we added task-supervised baselines and separated their role from the architecture-controlled sequence experiment.

Fine-tuned BGE-Reranker and Qwen-Embedding receive the same training evidence labels as CLESO. CLESO's ontology-derived structural targets remain method-specific. The historical-cohort comparisons in Table 3 are:

Method / evidence F1 (%)BioASQQASPERLegalBench-RAG
BGE: pretrained → fine-tuned26.40 → 16.6551.98 → 59.1022.88 → 37.34
Qwen-Embedding: pretrained → fine-tuned25.52 → 16.2453.49 → 58.7925.43 → 33.22
CLESO observation + direct Selector22.6657.2332.11
CLESO-CP26.9556.2529.98

These results show benchmark-dependent outcomes, rather than universal superiority over supervised rankers. Table 6 separately tests structural organization with architecture and visible features controlled. Supplementary accounting includes question-dependent features, refinement, Probe, package scoring, and generation/selection. On BioASQ/LegalBench-RAG, BGE requires 2.84×/12.98× CP's model workload, and Qwen3-Reranker requires 42.82×/143.65×. These are workload-proxy ratios, not speedups. CP is more expensive on short QASPER queues and than the evaluated SetR configuration across all three benchmarks.

Finally, the updated three-seed mean-F1 differences against the strongest baselines are +1.88/+0.20/+0.54 points. Q3 reports their paired uncertainty. Our supported contribution is learned budgeted organization and conditional workload savings, rather than uniformly better supervised accuracy or end-to-end latency.

For the questions:

Q1: Simple partitions and conventional supervised ranking

A1: The controls in W1 test precisely these alternatives: HEAD replaces structural training with global-head optimization, while RAW-Uniform replaces learned organization with equal-sized contiguous RAW partitions matched to PAPER's predicted interval count.

The downstream algorithms and budgets remain fixed, but the resulting observations and packages can change. Table 6 therefore measures the total effect of sequence organization within that pipeline, rather than a direct boundary effect with every realized package held constant. The conventional evidence-label rankers in Table 3 provide the separate overall supervised comparison. These additions appear in Sections 6.2 and 6.4.

Q2: Caching and question-dependent cost

A2: Corpus-side ontology annotations and query-independent representations can be cached across questions. Query-conditioned relevance scores are reusable only for the same query and fixed configuration. Query encoding, question-dependent feature assembly, refinement/decoding, Probe, package scoring, and proposal/selection run per question. Criterion search occurs during training. Inference uses the candidate evidence pool and available features without test labels, reference answers, or additional evidence. This separates training supervision from privileged test-time information.

Measured question-dependent feature processing is approximately 4/259/222 ms per question on QASPER/BioASQ/LegalBench-RAG. Figure 4 and Tables 14–15 supplement the historical runtime ledger with stage-level model workload and CPU components.

Q3: Comparable supervision and paired uncertainty

A3: Fine-tuned BGE-Reranker and Qwen-Embedding share the training evidence labels; HEAD additionally shares the architecture and visible features. Their historical, single-seed comparisons are separate from the updated three-seed main evaluation.

For the reported fixed F1 cohorts, CP minus the strongest three-seed mean-F1 baseline is:

Benchmark / comparator / F1 NSeed 13Seed 17Seed 29
BioASQ / RankGPT / 260+2.48 [−0.78, 6.16]+1.91 [−1.20, 5.35]+1.25 [−1.66, 4.55]
QASPER / SetR / 1,451−0.17 [−2.21, 1.84]−0.26 [−2.05, 1.58]+1.02 [−0.79, 2.82]
LegalBench-RAG / Qwen3-Reranker / 1,412+0.37 [−2.15, 2.92]+0.61 [−1.61, 2.77]+0.64 [−1.60, 2.88]

Entries are percentage-point differences with pointwise 95% CIs, from paired bootstrap resamples within each seed. QASPER is resampled by paper; the other benchmarks by question. Furthermore, Table 2 reports mean ± sample SD across the three seeded runs.

W1

The gain on BioASQ is small, and on QASPER RAW-Uniform is lower than RAW. It also lacks a control that applies the same decoder to the original scores. In addition, the F1 scores of the same method in Tables 2, 4, and 6 are different. The varying number of targets only explains why packing is adaptive, and Table 7 shows that the CP advantage mostly disappears at matched package size.

W2

The rankers fine-tuned with the same labels outperform CLESO-CP on QASPER and LegalBench-RAG. Workload proxies without end-to-end runtime under a common boundary do not resolve the cost concern.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.