acceptodds
Under review as a conference paper at ICLR 2027

AREA: Adaptive Relevance Evidence Allocation For Efficient LLM Reranking

Abstract

Large language model (LLM) rerankers provide strong relevance modeling but incur substantial inference cost because full-context inference propagates every input representation through all model layers. Existing efficient reranking methods typically rely on predefined compression configurations or externally selected operating points, despite substantial variation in where relevance evidence occurs and how much evidence different query–document pairs require. We formulate efficient reranking as adaptive relevance evidence allocation and introduce AREA, a pointwise reranker that learns both where full-resolution intermediate representations should be preserved and how much evidence each instance requires for deeper computation. AREA derives block-level localization supervision from the sensitivity of the relevance objective to compression, and instance-level budget supervision from counterfactual ranking feedback, without explicit token- or budget-level annotations. Under a fixed 50% retention ratio before ranking-guided optimization, AREA achieves 28.81 nDCG@10 on BRIGHT, compared with 27.72 for LTC. With adaptive budgeting, AREA reaches 30.21 nDCG@10 at an average retention ratio of 0.49, comparable to the separately optimized full-context variant at 30.10, while providing a controlled model-forward speedup. Consistent results on industrial search further demonstrate the effectiveness of task-derived localization and instance-specific budget allocation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
reranking, compression, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mfSdOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AREA, a pointwise LLM reranker that reduces inference cost by learning which intermediate representations to preserve at full resolution and how much to retain for each query–document pair. It derives localization supervision from a first-order approximation of the relevance-distillation objective’s sensitivity to compression, and learns retention budgets from counterfactual ranking feedback. Adaptive budgeting achieves performance comparable to the separately optimized full-context variant at an average retention ratio of 0.49, with a 1.33× speedup in controlled model-forward measurements.

Critical strengths
  • It is efficient to learn both which representations to preserve and how much computation each instance requires.
  • The method does not require explicit budget annotation, constructing supervision from relevance distillation and counterfactual ranking feedback
  • The evaluation, reporting ranking quality, retention ratios, and measured runtime, including wall-clock reranking time over complete candidate sets, supports the claim with some weakness (see below)
Critical weaknesses
  • Readibility can be improved. For example, terms such as block-level, document-level, instance-level, task-level, token-level, and budget-level are invented and used without propoer definition.

  • The baseline comparisons are weak.

  • In Table 1, A strong pointwise reranker, Retro*, is not included.
  • In Table 1, the batching strategy used for the reported wall-clock times is not specified. This matters because pointwise methods can exploit document-level parallelism, whereas listwise methods (ReasonRank) have a different batching structure.
  • To conclude that “adaptive” budget allocation is better, which is the paper’s key claim, budget allocation methods e.g. LTC should be the main baseline, and the statistical significance of the localization improvement over LTC should be evaluated. Relatedly, the external baselines in Table 1 cannot be attributed specifically to adaptive evidence allocation. AREA differs from the external baselines in backbone, training data, and supervision.
  • Further analysis of budget allocation behavior could be done. For example, analyzing whether low-budget assignments disproportionately hurt difficult queries or particular types of relevant documents.
Questions for authors
  • How robust are the learned localization and adaptive budgeting policies to sparse or noisy relevance supervision, given that both training signals are ultimately derived from relevance judgments?
  • Please check if the citations are correct. E.g. line 040, Xie et al., 2027
8 comments
AuthorsOpenReview ↗

Response to Reviewer mfSd – Comment: Readability & Batching Strategy and Runtime Comparison Fairness

Response to: Terminology and Readability

We thank the reviewer for pointing out the ambiguity in our terminology. We agree that the different granularities of supervision and computation allocation should be more clearly distinguished.

To address this concern, we have made the following revisions:

  1. Clarification in Section 3.1. We explicitly define an instance as a query–document pair when introducing instance-specific retention budgets.

  2. Unified terminology in Appendix A.3. We add a dedicated Terminology and Granularity paragraph that distinguishes token-level importance prediction, block-level importance scores and supervision, and instance-level budget allocation. We also clarify that document-level relevance refers to query–document relevance judgments, whereas task-level supervision denotes signals derived from the relevance-learning objective. Budget supervision refers to counterfactually derived retention targets rather than a separate annotation granularity.

  3. Consistent wording throughout the manuscript. We replace potentially ambiguous expressions in the Abstract and Introduction, such as "token- or budget-level annotations," with more explicit descriptions, including "token-importance annotations" and "target retention budgets."

These changes clarify the relationship between the available relevance judgments, task-derived supervision, and learned allocation decisions while preserving the concise presentation of the main method.

Response to: Batching Strategy and Runtime Comparison Fairness

We thank the reviewer for raising this important concern. We agree that the different batching and candidate organization strategies of pointwise and listwise rerankers should be considered when interpreting wall-clock efficiency.

Batching protocol. For external baselines, we use vLLM continuous batching, with batch sizes and candidate grouping following their official inference implementations. All methods are evaluated using eight NVIDIA A800 80GB GPUs on the same BM25 top-100 candidate sets.

Evaluation scope. Table 1 compares system-level ranking effectiveness and efficiency under each method's inference configuration, rather than strictly matched model-forward latency. We have clarified this distinction in Section 4.2 and supplemented the batching details in Appendix A.4.

Controlled efficiency comparison. Appendix C.1 provides model-forward measurements under a matched HuggingFace implementation with batch size one. Under these controlled conditions, AREA (Adaptive) achieves a 1.33× speedup over AREA (Full), reducing latency from 89.30 ms to 67.21 ms per query–document pair.

Together, these evaluations characterize both end-to-end candidate-set reranking efficiency and the computational savings achieved by AREA's adaptive allocation mechanism.

AuthorsOpenReview ↗

Response to Reviewer mfSd – Comment: Attribution of Improvements to Adaptive Evidence Allocation

Response: Attribution of Improvements to Adaptive Evidence Allocation

We thank the reviewer for this important observation. We agree that comparisons with independently trained external rerankers cannot isolate the contribution of adaptive evidence allocation, as these methods differ in backbone architectures, training data, and optimization objectives.

Additional competitive baseline. We have added RETRO*-8B to Table 1, evaluated on the same original-query BM25 top-100 candidate sets using single-sample inference (k = 1), without additional test-time sampling. RETRO* achieves 24.88 nDCG@10 on BRIGHT, macro-averaged over the 12 domains. The inference configuration is specified in Section 4.1.

Our experimental evaluation therefore distinguishes system-level effectiveness–efficiency comparisons from controlled component-level evaluations:

  1. Evidence Localization (Section 4.3 and Appendix B.1). We compare AREA with LTC and random block selection using the same pre-RL reranker, compression layer, and retention budget. At 50% retention, AREA achieves 28.81 nDCG@10 on BRIGHT, compared with 27.72 for LTC. This comparison isolates the effectiveness of task-aligned evidence localization without introducing differences in the underlying reranker or ranking-guided RL.

  2. Adaptive Budgeting (Section 4.4). Fixed, Cumulative, and Adaptive variants share the same pre-RL initialization and are independently optimized under matched RL settings. At approximately 50% retention, Adaptive achieves 30.21 nDCG@10, compared with 29.21 for Fixed and 29.33 for retention-matched Cumulative. The Shuffled Budget control, which permutes predicted budgets across instances using the same Adaptive checkpoint, achieves 29.36, further supporting the importance of instance-specific budget assignment.

We have revised Section 4.2 to distinguish the system-level comparisons in Table 1 from the controlled component evaluations, and Section 4.3 to explicitly identify LTC as the primary localization baseline.

We agree that the external baselines establish the overall competitiveness of AREA rather than attributing their performance differences solely to adaptive allocation. The contributions of evidence localization and adaptive budgeting are instead supported by the controlled comparisons above.

AuthorsOpenReview ↗

Response to Reviewer mfSd – Comment: Statistical Significance of Evidence Localization over LTC

Response: Statistical Significance of Evidence Localization over LTC

We thank the reviewer for emphasizing the importance of comparing AREA against established compression methods and statistically validating the localization improvement.

We agree that LTC is a particularly relevant baseline for evaluating where intermediate representations should be preserved. Accordingly, our localization experiment compares AREA with LTC under controlled conditions: both methods use the same pre-RL reranker, compression layer, and fixed retention ratio, without ranking-guided RL.

To further assess statistical reliability, we have added a dedicated analysis in Appendix B.4, including a new table reporting domain-stratified paired bootstrap results. We perform 10,000 query-level bootstrap resamples over all 1,384 BRIGHT queries, independently within the 12 domains, and recompute the macro-average nDCG@10 difference.

At 50% retention, AREA Localization achieves 28.81 nDCG@10, compared with 27.72 for LTC, yielding an improvement of +1.09 points. The corresponding 95% bootstrap confidence interval is [+0.3798, +1.9001], excluding zero.

This result provides statistical evidence that AREA's task-aligned localization improves ranking effectiveness over LTC under matched compression and reranker settings, independently of subsequent adaptive budget optimization.

Reviewer_qBJbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces Adaptive Relevance Evidence Allocation (AREA), an efficient pointwise LLM reranker. AREA improves inference efficiency by compressing representations at an intermediate layer. A lightweight importance predictor identifies which blocks should retain full-resolution representations. The remaining blocks are pooled before being passed to later layers.

With an adaptive budget, AREA retains 49% of the blocks on average and achieves an nDCG@10 of 30.21. This is comparable to the separately optimized full-context variant, which scores 30.10. AREA also delivers a 1.33× speedup in the model’s forward pass under controlled conditions.

Critical strengths
  1. The paper frames efficient pointwise LLM reranking as an adaptive evidence allocation problem. AREA decides both where to keep full-resolution representations and how much evidence each query–document pair needs in deeper layers.

  2. The method derives two levels of supervision from the ranking objective at very little computational cost. Block-level supervision comes from how sensitive the relevance objective is to compression. Instance-level budget supervision comes from counterfactual ranking feedback. This avoids the need for explicit token-level or budget labels and makes the approach practical.

Critical weaknesses
  1. The writing is hard to follow. Abbreviations need clearer definitions, citations, or brief explanations. For example, LTC appears several times in the abstract and introduction, but its source is not given. The paper also compares three main settings—“Full,” “Cumulative,” and “Adaptive”—yet their definitions are difficult to find.

  2. Reproducibility is a concern. AREA has many hyperparameters, and key parts of the training code are missing. This makes it difficult to reproduce the results.

Questions for authors
  1. BRIGHT already has a long-context version. How does the paper’s BRIGHT-Long differ from the original BRIGHT-Long?

  2. The nDCG scores on the industrial dataset reach 92–95. Is this dataset relatively easy? Also, are the results statistically significant with only 300 test queries? Industrial evaluations usually use much larger test sets.

4 comments
AuthorsOpenReview ↗

Response to Reviewer qBJb – Comment 1 & Comment 2

Response to Comment 1: Presentation and Terminology

We thank the reviewer for pointing out these presentation issues. We agree that several terms and experimental configurations should be introduced more clearly to improve readability.

We have made the following revisions:

  1. LTC terminology and citation. We expand LTC as Layer-wise Token Compression in the Abstract and provide the corresponding citation (Zhuang et al., 2026b) at its first explicit mention in the Introduction.

  2. Clear definitions of AREA variants. In Section 4.1, we now explicitly define Full (no intermediate compression), Cumulative (retention based on a predefined cumulative importance threshold), and Adaptive (instance-specific budget prediction). We also clarify that the variants share the same pre-RL initialization but are optimized separately under matched RL settings.

  3. Distinguishing Fixed from Full. In Section 4.4, we clarify that Fixed applies a uniform retention ratio of 0.5 to all query–document pairs, whereas Full disables intermediate compression.

We have also reviewed the terminology and abbreviations throughout the manuscript to improve clarity and consistency. These revisions make the experimental configurations easier to understand without requiring readers to consult the appendix.

Response to Comment 2: Reproducibility

We thank the reviewer for emphasizing the importance of reproducibility. We agree that the multi-stage training procedure and implementation details should be sufficiently documented to support independent reproduction.

Algorithmic and implementation details. We have substantially expanded the methodological documentation. Algorithm 1 provides the complete four-stage training procedure, including cold-start SFT, relevance distillation, evidence localization, and ranking-guided adaptive budgeting. It explicitly describes the parameter-freezing strategies, construction of localization targets, directional counterfactual budget search, and separate optimization of the reranker and budget predictor.

Furthermore, Tables 5–8 provide detailed training configurations, budget-learning hyperparameters, training computation costs, and module architectures. Appendices A.2–A.4 further specify the training procedure, compression implementation, inference settings, and efficiency measurement protocols. Together, these materials provide the principal algorithmic steps, hyperparameters, and implementation specifications needed to guide an independent reproduction of AREA.

Data construction and evaluation. We have included the BRIGHT-Recon document reconstruction and relevance-label mapping scripts in the supplementary materials, enabling reproduction of our reconstructed benchmark. The BM25 candidate construction and nDCG evaluation follow the official BRIGHT implementation and evaluation protocol.

Code availability. Due to company proprietary restrictions, we cannot commit to publicly releasing the core training and inference code. To address reproducibility concerns within these constraints, we have provided detailed algorithmic pseudocode, training configurations, implementation specifications, and the BRIGHT-Recon construction scripts described above.

We appreciate the reviewer's suggestion and believe these additions substantially improve the transparency and reproducibility of our work.

AuthorsOpenReview ↗

Response to Reviewer qBJb – Q1: Differences from the Official BRIGHT Long-Document Benchmark

Response to Q1: Differences from the Official BRIGHT Long-Document Benchmark

We thank the reviewer for raising this question. We clarify that our BRIGHT-Long is a reconstructed longer-input evaluation setting, rather than the official BRIGHT long-document benchmark. To avoid confusion, we have renamed it BRIGHT-Recon throughout the manuscript.

1. Differences and motivation. The official BRIGHT long-document setting uses original full-length documents, which contain 17,256 tokens on average according to our analysis. This substantially exceeds the training input lengths of AREA (e.g., 2,560 tokens for Budget/RL) and several compared rerankers. For example, ERank uses a maximum prompt length of 1,024 tokens during RL training. Direct evaluation on these documents may therefore introduce substantial train–test length mismatch and heterogeneous truncation effects. Aggressive truncation can also remove relevance-bearing evidence while retaining the original relevance labels, potentially introducing inconsistencies between the evaluated text and its annotations.

To mitigate these issues, BRIGHT-Recon constructs moderately long candidates by merging consecutive source-document chunks where applicable, targeting approximately 1,024 tokens, while retaining suitable self-contained documents in the remaining domains. It covers all 12 BRIGHT domains and evaluates up to 50 BM25 candidates per query. The resulting benchmark contains 1,193 queries and 58,409 query–document pairs (Appendix D.2).

2. Reliability of transferred relevance labels. BRIGHT uses binary relevance judgments, and our reconstruction follows a consistent mapping rule: a reconstructed document is considered relevant if it contains at least one originally relevant chunk. This avoids the ambiguity of aggregating graded relevance scores. We additionally conducted a manual sanity check on 50 randomly sampled query–document pairs, finding all transferred labels reasonable under the intended relevance criteria.

3. Evaluation scope. We acknowledge that document reconstruction and label transfer may alter relevance granularity. Therefore, BRIGHT-Recon is intended to evaluate the effectiveness–efficiency trade-off under controlled longer-input conditions, rather than replace the official benchmark for naturally occurring full-length documents. We have clarified this distinction in Appendix D.2 and explicitly acknowledged the limitation in the Conclusion.

AuthorsOpenReview ↗

Response to Reviewer qBJb – Q2: Industrial Dataset Difficulty and Statistical Significance

Response: Industrial Dataset Difficulty and Statistical Significance

We thank the reviewer for raising concerns about the relatively high nDCG@10 scores and the statistical reliability of our industrial evaluation.

1. Interpretation of the high nDCG@10 scores. The industrial benchmark differs from BRIGHT in candidate-set construction and evaluation setting. BRIGHT reranks the top-100 BM25-retrieved candidates, whereas our industrial benchmark contains an average of 5.76 candidates per query. For the industrial benchmark, we first construct each query's candidate set and then have professional assessors annotate every query–document pair within it. We compute nDCG@10 using linear gains from the four-level relevance labels, with IDCG obtained by sorting the same annotated candidate set. For queries with fewer than ten candidates, the metric is computed over all available candidates. These scores therefore measure ranking quality within the predefined candidate sets and do not measure retrieval recall over the full document corpus. The relatively high absolute nDCG@10 scores should be interpreted within this evaluation setting, and direct comparisons with BRIGHT's absolute scores are inappropriate.

2. Dataset construction and difficulty. The industrial benchmark was constructed from real-world production search logs, combining randomly sampled queries with medium- and long-tail queries. The collected queries also include cases where deployed ranking models exhibit poor performance, ensuring that the evaluation covers challenging real-world search scenarios beyond frequently occurring queries. All query–document pairs were manually annotated by professional relevance assessors using four relevance levels (Excellent, Good, Fair, and Irrelevant). The relatively small test size reflects the cost of professional relevance annotation rather than an intentional simplification of the ranking task.

As an additional reference, we evaluate Qwen3.5-397B-A17B without task-specific fine-tuning, using pointwise relevance prediction on the same 300-query industrial test set. The model independently assigns a relevance score from 0 to 3 to each query–document pair and achieves 82.71 nDCG@10 under the same evaluation protocol. This result provides additional evidence that high ranking effectiveness is not trivially achieved on the benchmark, even with a substantially larger general-purpose model. We have included this reference evaluation in Appendix D.1.

3. Statistical significance. To address the concern regarding the 300-query test size, we conducted an additional query-level paired bootstrap analysis with 10,000 resamples, preserving complete candidate groups and reporting 95% confidence intervals for both nDCG@10 and nPNR. The results are:

  • Adaptive vs. Fixed: ΔnDCG@10 = +1.66, 95% CI [+1.19, +2.58]; ΔnPNR = +0.0257, 95% CI [+0.0103, +0.0548].
  • Adaptive vs. Cumulative (kx = 0.6): ΔnDCG@10 = +1.16, 95% CI [+0.97, +1.54]; ΔnPNR = +0.0130, 95% CI [+0.0093, +0.0167].
  • Adaptive vs. Shuffled Budget: ΔnDCG@10 = +0.99, 95% CI [+0.72, +1.29]; ΔnPNR = +0.0209, 95% CI [+0.0125, +0.0300].

All six confidence intervals exclude zero, providing statistical support for AREA's improvements over the compared budget allocation strategies under query-level resampling.

We have added these results to Appendix C.4 and clarified the corresponding findings in Section 4.4. While the limited number of industrial queries remains a constraint on broader generalization, the targeted evaluation design, four-level relevance annotations, and additional statistical analysis support the validity of the reported comparisons.

Thank you to the authors for their response. I will maintain my original score.

Reviewer_yxF7OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AREA, an efficient pointwise LLM reranker that learns both evidence localization and instance-specific retention budgets. It derives block-level supervision from relevance-objective sensitivity and budget supervision from counterfactual ranking feedback. Experiments on BRIGHT and additional retrieval benchmarks demonstrate improved ranking effectiveness under reduced computation.

Critical strengths
  1. Meaningful methodological contribution. The combination of task-aligned evidence localization and ranking-guided adaptive budgeting provides a principled approach to allocating computation without explicit token-level annotations.

  2. Convincing controlled experiments. AREA outperforms LTC under matched retention budgets. The adaptive budgeting experiments include fixed, cumulative, and shuffled-budget controls. Multi-seed results, paired bootstrap confidence intervals, and controlled latency measurements provide strong empirical support.

Critical weaknesses
  1. Insufficient accounting of training complexity.

The framework requires distillation-based localization supervision, counterfactual budget searches, and ranking-guided optimization. Although inference efficiency is carefully evaluated, the additional training computation required for learned allocation is not sufficiently quantified relative to simpler alternatives.

  1. Limitations of long-document evaluation.

BRIGHT-Long is constructed by merging or selecting documents and transferring existing relevance annotations. This may change relevance granularity and does not fully establish effectiveness on naturally occurring long documents.

Questions for authors
  1. Can the authors quantify the additional training computation required for localization and counterfactual budget supervision relative to fixed and cumulative allocation?

  2. Are results available on naturally occurring long-document benchmarks without reconstructed relevance annotations?

2 comments
AuthorsOpenReview ↗

Response to Reviewer yxF7 – Q1: Training Computational Cost

Response to Q1: Additional Training Computation

We thank the reviewer for highlighting the importance of training computation. We have expanded the cost analysis to include the complete training pipeline and comparisons against both fixed and cumulative budget allocation.

Shared training cost. For the public AREA model, 9B student SFT, 27B teacher SFT, on-policy relevance distillation, and evidence localization require 51.60, 165.60, 267.20, and 7.60 GPU-hours, respectively, totaling 492.00 GPU-hours. These stages are shared across the allocation variants and are counted only once per training pipeline.

RL-stage and total training costs. Under matched 15-epoch RL settings on eight NVIDIA A800 GPUs, Fixed, Cumulative (kx=0.6), Cumulative (kx=0.8), and Adaptive require 262.13, 265.20, 271.20, and 264.93 GPU-hours, respectively. Including the shared stages, their complete model-training costs are 754.13, 757.20, 763.20, and 756.93 GPU-hours, respectively.

Relative to Fixed, Adaptive incurs an additional 2.80 GPU-hours (+1.07%) during RL, corresponding to only +0.37% of the complete Fixed training cost. Compared with Cumulative (kx=0.6), Adaptive requires 0.27 fewer GPU-hours (−0.10%) during RL; compared with Cumulative (kx=0.8), it requires 6.27 fewer GPU-hours (−2.31%). These comparisons show that adaptive allocation incurs limited additional training cost under matched settings.

Counterfactual budget search. The reported Adaptive Budget/RL cost includes counterfactual search. Our implementation reuses intermediate hidden states at the compression layer, and each additional budget probe requires only a partial forward through subsequent backbone layers. Directional search evaluates an average of 7.98 budget configurations per candidate, including the initial Budget Head proposal, compared with 16 operating points for exhaustive grid evaluation (Appendix C.5).

We have updated Table 7 and Appendix A.2 to report the training-stage breakdown and full costs, and Section 4.4 to clarify the incremental overhead relative to Fixed and retention-matched Cumulative allocation.

AuthorsOpenReview ↗

Response to Reviewer yxF7 – Q2: Long-Document Evaluation

Response to Q2: Evaluation on Naturally Occurring Long Documents

We thank the reviewer for raising this important concern. We agree that our reconstructed longer-input evaluation does not fully establish effectiveness on naturally occurring full-length documents with native relevance annotations.

1. Motivation for reconstruction.

Our analysis of the original BRIGHT long-document corpus shows an average document length of 17,256 tokens, substantially exceeding the training input lengths of AREA (3,072 tokens for localization and 2,560 tokens for Budget/RL). Several compared rerankers are also trained under much shorter input configurations. For example, ERank uses a maximum prompt length of 1,024 tokens during RL training.

Directly evaluating these rerankers on the original full-length documents may therefore introduce substantial train–test length mismatch and heterogeneous truncation effects. More importantly, aggressive truncation can remove the actual relevance-bearing evidence while retaining the original relevance judgment, creating inconsistencies between the evaluated text and its assigned label.

To mitigate these issues, we construct moderately long candidates by merging source-document chunks where applicable, targeting approximately 1,024 tokens, while retaining suitable self-contained documents in the remaining domains. This provides a controlled longer-input evaluation across all 12 BRIGHT domains.

2. Reliability of transferred relevance annotations.

We acknowledge the reviewer's concern that document reconstruction may alter relevance granularity. Nevertheless, our construction has several properties that help mitigate this issue.

First, BRIGHT employs binary relevance judgments (relevant/non-relevant) rather than graded relevance scores. Our mapping follows an existential relevance criterion: a reconstructed document is labeled relevant if it contains at least one originally relevant chunk; documents composed entirely of non-relevant chunks retain a non-relevant label. This avoids the additional ambiguity of aggregating graded relevance judgments. While adding surrounding context may dilute relevant evidence or occasionally change semantic relevance, binary mapping provides a simple and consistent labeling rule.

Second, we conducted an additional manual sanity check by randomly sampling 50 query–document pairs from the reconstructed evaluation set. All 50 transferred relevance labels were judged reasonable under the intended relevance criteria. Although this small-scale check cannot eliminate all annotation uncertainty, it provides supporting evidence for the reliability of our construction.

3. Clarification of the evaluation scope.

Under this controlled setting, AREA (Adaptive) reduces reranking time by 29.7% compared with AREA (Full), while achieving comparable nDCG@10 (37.64 vs. 38.06; Table 3).

We agree that this evaluation does not replace naturally occurring long-document benchmarks with native relevance judgments. We have renamed our reconstructed benchmark BRIGHT-Recon, clarified the binary relevance-mapping procedure and manual validation in Appendix D.2, and explicitly acknowledged the limitations of reconstructed longer-input evaluation in Section 5 (Conclusion).

We appreciate the reviewer's suggestion and believe these clarifications better define both the motivation and scope of our longer-input evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.