acceptodds
Under review as a conference paper at ICLR 2027

Relevance Is Not Variance: Local Fisher-Spectral Adaptation for Dense Retrieval

Abstract

Retrieval-augmented generation (RAG) relies on relevant external evidence to ground language model responses. Retrieving this evidence remains challenging when a fixed embedding space must serve queries with different information needs. Directions that capture substantial variation in that space may offer little help in distinguishing relevant evidence from distractors. We study how relevance feedback can guide query adaptation while keeping the encoder and document index unchanged. We propose Fisher-Spectral Adaptation (FISA), a supervised adapter that uses the initial retrieval results to identify locally discriminative directions and refine the query for a second search. Experiments with five frozen embedding models on four BEIR tasks show a 3.63-point gain in macro nDCG@10 (8.58% relative) over the strongest deterministic baseline, with improvements in 16 of 20 encoder–dataset comparisons. The gains are stable across 20 training seeds, while evaluation on an unseen domain reveals limits to transfer. These results demonstrate the potential of adapting retrieval geometry to query-specific evidence.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
dense retrieval, query adaptation, relevance feedback, retrieval geometry, frozen embeddings

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_M5eoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Fisher-Spectral Adaptation (FiSA), a supervised query adaptation method for dense retrieval. FiSA consists of three components: (1) query-dependent coordinate reweighting, (2) a local Fisher-Spectral correction based on the initially retrieved documents, and (3) a confidence gate that controls the query update. The adapted query is then used for a second full-corpus retrieval, while the document embeddings and index remain unchanged.

Experiments are conducted on four BEIR datasets using five frozen embedding models. The results show significant improvements over several baselines.

Critical strengths
  • Promising in-domain experimental results. FiSA achieves strong improvements on several datasets, especially in-domain ones. The method is evaluated with five different embedding models, showing that it can be applied to different frozen retrievers.
Critical weaknesses
  • Query-local PCA basis seems to be not siginificantly better than global PCA bases. A central motivation of this paper is that a query-specific local coordinate system can better capture relevance than a global coordinate system. However, the ablation results in Table 4 show that replacing the query-local PCA basis with a global PCA basis only reduces average nDCG@10 from 45.93 to 45.79.

  • Much of the improvement comes from supervised coordinate reweighting, rather than the Fisher-Spectral correction. According to Table 4, the Dimension Only variant already achieves 44.79 macro nDCG@10, compared with 45.93 for the full FiSA model. This suggests that a large portion of the improvement comes from supervised query embedding adaptation.

  • The Fisher-Spectral component uses learned positive and negative attention distributions trained with relevance labels. A natural baseline would be to use these same attention distributions to compute positive and negative document centroids, and directly update the query using a Rocchio-style correction. This would help isolate the contribution of the Fisher-Spectral computation from that of learned relevance feedback.

  • The improvements are concentrated on in-domain datasets, and cross-domain generalization remains limited. FiSA is trained on NFCorpus, SciFact, and FiQA, and evaluated on the test sets of these three datasets and one unseen dataset, ArguAna. Compared with ECLIPSE-PRF, FiSA improves average nDCG@10 by 12.04 points on SciFact, but only 1.58 and 1.41 points on NFCorpus and FiQA, respectively. This suggests that the overall improvement of 3.63 points is largely driven by SciFact. On the only unseen dataset, ArguAna, FiSA underperforms ECLIPSE-PRF on four out of five encoders. This raises concerns about the generalization ability of the learned query adaptation mechanism.

  • Missing end-to-end efficiency comparison. FiSA requires an additional full-corpus retrieval, a per-query eigendecomposition, and several neural network modules. The additional inference cost is not fully evaluated and compared.

Questions for authors
  • Can the authors compare FiSA with a supervised Rocchio-style method using the same learned positive and negative attention distributions? This would help clarify the contribution of the Fisher-Spectral correction.
1 comment
AuthorsOpenReview ↗

With the submitted model’s learned attention held fixed, Fisher minus centroid is −0.083 points under the original first pass and +0.253 under the filtered first pass. The revised closed-form method gains 0.344 points over its matched centroid. We also report transfer results and end-to-end latency.

Response to Reviewer M5eo

Thank you for proposing the same-attention centroid comparison. We ran that intervention on the submitted neural model and added the corresponding matched-input control for the revised closed-form formulation. The revision removes neural adapter training and uses two fixed linear maps fitted offline on source relevance labels.

1. Same learned attention, centroid update

We use the original BGE-base checkpoint with seed 2025. For each first-pass protocol, the Fisher and centroid arms share the checkpoint, candidates, and learned positive and negative attention. The centroid direction uses the difference between the two attention-weighted document means.

First-pass protocolNeural FisherSame-attention centroidDifference [paired 95% interval]
Original58.13458.216−0.083 [−0.316, 0.156]
Filtered57.09856.845+0.253 [0.025, 0.482]

Scores are nDCG@10 points. The direction effect changes sign across the two first-pass protocols in this single-checkpoint replay.

For the revised method, the matched centroid shares both source-fitted maps, the initial candidates, positive and negative feedback weights, and step 0.35. Across all 30 encoder–collection pairs, closed-form FiSA scores 56.433 versus 56.089, a difference of +0.344 [0.213, 0.477]. With an independently step-tuned centroid, the difference is +0.190 [0.090, 0.295] (Figure 3, Table A4).

2. Attribution to local geometry and calibration

In the submitted model, the local/global basis comparison is +0.149 [−0.029, 0.327], and Dimension Only already scores 44.788 versus 45.935 for the full model. In the revision, removing the initial map costs 1.639 points and removing lexical feedback costs 1.439 in separate ablations. The matched comparison above measures the smaller increment due to the Fisher direction with those inputs held fixed. The revised method computes it by a regularized candidate-space solve and removes the learned spectral controller and eigendecomposition (Section 3, Table A4).

3. Transfer beyond SciFact

We added SciDocs and Quora as evaluation-only collections alongside ArguAna. Closed-form FiSA exceeds ECLIPSE-PRF in 14/15 encoder–collection comparisons across those three domains, with means of 56.644 and 55.313. Excluding SciFact from the average gives 50.821 versus 49.062. On ArguAna, FiSA exceeds ECLIPSE-PRF for four encoders and Raw Dense for one. This difference makes the encoder-level table particularly relevant (Table 1, Table A3).

4. End-to-end cost and presentation

We measured tokenization through final Top 1000 on the same BGE-base CPU setup with two threads, batch size one, and 100 queries per collection repeated three times. The timer includes both FiSA searches. FiSA P50/P95 is 13.85/18.61 ms on SciFact and 33.47/39.99 ms on Quora, versus 13.46/18.44 ms and 32.35/35.93 ms for ECLIPSE-PRF. All four methods are reported on six collections in Table A10 and Figure 4. These measurements apply to the revised method. We also rewrote Section 3 in execution order and moved the derivation and numerical details to Appendix A.

Reviewer_L6pkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript introduces Fisher-Spectral Adaptation (FISA), a supervised query adaptation methodology designed to improve Retrieval-Augmented Generation (RAG) by dynamically refining queries while keeping the document index and embedding encoder frozen. The framework operates in a two-pass retrieval system using three sequential stages: query-aware coordinate reweighting, a local Fisher-spectral update that leverages first-pass retrieval candidates to find locally discriminative directions, and a confidence-gated residual step that limits the query update before a second full-corpus search. Empirically, FISA yields a 3.63-point gain (8.58% relative improvement) in macro nDCG@10 over the strongest deterministic baseline across five frozen encoders and four BEIR tasks.

Critical strengths

Practical Modularity and Non-Destructive Integration: A major operational advantage of FISA is its ability to extract RAG performance improvements without requiring downstream re-indexing or encoder retraining. Because it generates a single refined vector that is passed to a standard maximum-inner-product search, it acts as a modular adapter that is highly practical to deploy within existing, large-scale dense retrieval systems.

Critical weaknesses

Confounded Attribution of Gains Due to Unmatched Supervision: FISA achieves its gains by training a neural network on labeled source-domain data, yet it compares itself against deterministic, unsupervised baselines (Raw Dense, SPECTEMP-256, and ECLIPSE-PRF) that do not exploit these supervised learning signals. While the authors acknowledge this limitation in the text, the absence of a comparably supervised baseline—such as a naive Multilayer Perceptron (MLP) query adapter or supervised Rocchio framework—makes it impossible to definitively attribute the performance boost to the specific Fisher-spectral architecture rather than the raw introduction of supervised relevance labels.

Questionable Necessity of the Local Spectral Basis: The paper heavily anchors its contribution on the theoretical need for a "query-local view of retrieval geometry". However, the ablation study explicitly undermines this premise: replacing the dynamically computed local basis with a static, global PCA basis results in a negligible performance gap of only 0.15% in average nDCG@10. Furthermore, the global basis actually outperformed the localized approach on the unseen ARGUANA dataset. This strongly suggests the computational overhead of generating query-specific local spectral bases may not be practically justified.

Insufficient Validation of Domain Generalization: FISA is trained on a mixture of three datasets (NFCORPUS, SCIFACT, FIQA) and tested on only one unseen domain (ARGUANA), where it noticeably struggles to transfer its learned geometries and underperforms the ECLIPSE-PRF baseline. Because query adaptation behavior is highly dependent on dataset characteristics, a single hold-out domain is insufficient. Leave-One-Dataset-Out experiments across the three source domains are necessary to prove the adapter learns generalizable retrieval geometries rather than dataset-specific shortcuts.

Questions for authors

To isolate the structural benefits of the FISA architecture from the benefits of its supervised training, could you include a comparison against a structurally simple supervised baseline (e.g., a supervised MLP query adapter or supervised Rocchio)?

Given that the Global Basis ablation performs within 0.15% of the Full FISA architecture and actually surpasses it on the unseen ARGUANA domain, what is the compelling justification for the added algorithmic complexity of calculating a query-specific local spectral basis?

The adapter's performance is intrinsically linked to the datasets it trains on. Will you provide Leave-One-Dataset-Out cross-validation results across NFCORPUS, SCIFACT, and FIQA to clarify how much of the performance relies on the specific source mixture?

1 comment
AuthorsOpenReview ↗

Sharing both fitted maps and feedback with a simpler centroid control isolates a 0.344-point Fisher gain in the closed-form revision. A separate three-domain holdout pilot for the submitted neural model shows the largest drop on SciFact, from 84.885 to 74.358.

Response to Reviewer L6pk

Thank you for asking us to control for supervision and test source-domain dependence. The revision replaces the submitted neural adapter with closed-form FiSA. Its encoder and document index stay fixed. Both linear calibration maps still use source relevance labels for offline fitting.

1. A simpler comparator with the same supervision

For the revised method, we use a centroid update that shares both fitted maps, the candidates, positive and negative weights, and the 0.35 update step. The sole intervention is to use the centroid direction in place of the Fisher mixture. Closed-form FiSA scores 56.433 versus 56.089 for this matched comparator across five encoders and six collections. The difference is +0.344 nDCG@10 points, with paired 95% interval [0.213, 0.477]. The independently step-tuned centroid scores 56.243, leaving +0.190 [0.090, 0.295]. The matched comparator uses the same label-fitted inputs as FiSA (Figure 3, Table A4).

The submitted neural model has two complementary controls. The five-seed Dimension Only adapter scores 44.788, compared with 45.935 for full neural FiSA. A single BGE-base checkpoint replay also applies a centroid update using the same learned positive and negative attention. The Fisher-minus-centroid differences are −0.083 with the original first pass and +0.253 with the filtered first pass. This replay keeps the trained attention fixed.

2. Local geometry and its cost

The submitted local/global difference is +0.149 [−0.029, 0.327]. The interval includes zero, and the global variant is stronger on ArguAna. In a separate single-checkpoint intervention that fixes centering and attention, the local basis gains 0.580 and 0.698 points under the two first-pass protocols. The revised paper narrows its claim to the measured increment of the local Fisher update over the matched centroid.

The revised update computes regularized within-group scatter from the candidates and uses a 128-by-128 Cholesky solve to obtain the Fisher direction. The implementation removes the per-query eigendecomposition and still performs two searches. Measured FiSA P50/P95 is 13.85/18.61 ms on SciFact and 33.47/39.99 ms on Quora, versus 13.07/18.07 and 22.48/26.60 ms for Raw Dense, on the same BGE-base CPU setup (Appendix A, Section 4.4, Figure 4).

3. Leave-one-dataset-out source dependence

We ran all three requested source holdouts as a pilot for the submitted neural architecture using BGE-base and seed 2025. Each target is excluded from both fitting and checkpoint selection. Scores are nDCG@10 points.

Held-out targetAll-source FiSAHeld-out FiSAHeld-out same-attention centroid
NFCorpus40.16838.34639.378
SciFact84.88574.35874.135
FiQA41.60039.55438.420

SciFact drops by 10.527 points, the largest decrease in this pilot. In the closed-form evaluation, ArguAna, SciDocs, and Quora are held out of map fitting. Their mean is 56.644 versus 55.313 for ECLIPSE-PRF. Refitting the two closed-form maps under each source holdout remains open (Table 1).

Reviewer_RCUNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Fisher-Spectral Adaptation (FISA), a supervised adapter which leverages the retrieval results in the first stage and refines query for the second retrieval. Through combining query-conditioned coordinate reweighting, local Fisher-spectral correction, and confidence gating, FISA adapts the query while keeping the encoder and document index frozen. Experiments with five embedding models on four BEIR datasets show a 3.63-point improvement in macro nDCG@10 over the strongest evaluated baseline, with gains in 16 of 20 encoder–dataset comparisons.

Critical strengths
  1. This paper proposes a query adapter which combines a local spectral basis with learned positive and negative evidence, while using confidence gating to regulate the update. The ablation results support the benefit of candidate feedback beyond coordinate reweighting.
  2. This paper conducts experiments and ablation studies with five embedding models on four BEIR datasets. The results demonstrate the effectiveness and stability of the proposed method.
Critical weaknesses
  1. Although FISA achieves better results on NFCorpus, SciFact, and FiQA, the adapter is trained on these three domains. On the unseen ArguAna dataset, it underperforms ECLIPSE-PRF for four of the five encoders, raising concerns about its generalization ability. Moreover, most of the aggregate gain comes from SciFact, which weakens the evidence for consistent improvements across domains.
  2. This paper does not provide a consistent end-to-end efficiency evaluation. Since FISA requires local spectral decomposition and an additional full-corpus search, the computational cost of its performance gains remains unclear.
Questions for authors
  1. Have the authors conducted experiments on more unseen datasets to evaluate the generalization ability of the proposed method?
  2. What are the inference latency of the proposed method compared with the baselines under the same experimental setup?
1 comment
AuthorsOpenReview ↗

With SciDocs and Quora added, closed-form FiSA outperforms ECLIPSE-PRF in 14 of 15 comparisons across three evaluation-only collections. Excluding SciFact, the mean is 50.821 versus 49.062. We also measure all four methods from tokenization to final ranking under the same CPU setup.

Response to Reviewer RCUN

Thank you for asking for broader transfer evidence and a matched latency benchmark. The revised manuscript replaces the submitted neural adapter with closed-form FiSA, retaining the frozen encoder, fixed index, and two retrieval passes. Its two linear maps are fitted offline on source relevance labels from NFCorpus, SciFact, and FiQA.

1. Evaluation on additional domains

We added SciDocs and Quora and evaluated all five frozen encoders. These two collections and ArguAna are evaluation-only in this protocol. The table reports mean nDCG@10 points across the five encoders.

Evaluation-only collectionRaw DenseECLIPSE-PRFClosed-form FiSA
ArguAna58.87557.64059.764
SciDocs20.43220.83621.669
Quora87.91487.46488.500
Mean over the three collections55.74155.31356.644

FiSA exceeds ECLIPSE-PRF in 14 of 15 encoder–collection comparisons across these evaluation-only domains, including 4 of 5 on ArguAna and 5 of 5 on each new collection. The ArguAna result depends on the comparator and encoder. Against Raw Dense, four encoders lose ground and E5-base gains 7.917 points. We report the full encoder-level results in Table A3.

SciFact still contributes substantially to the overall average. Across all six collections, FiSA scores 56.433 versus 52.625 for ECLIPSE-PRF. After removing SciFact from the reported average, FiSA scores 50.821 versus 49.062 for ECLIPSE-PRF and 49.136 for Raw Dense. This sensitivity check leaves the fitted maps fixed (Table 1).

2. End-to-end latency under the same setup

We used the same frozen BGE-base encoder, Mac M5 Pro CPU, two PyTorch threads, batch size one, and 100 queries per collection with three repetitions per query. The timer starts at tokenization and stops at the final Top 1000. For FiSA it includes the encoder, calibration and feedback, and both full-corpus searches. All measurements start with a resident model and index.

DatasetRaw DenseECLIPSE-PRFRocchioClosed-form FiSA
SciFact13.07 / 18.0713.46 / 18.4413.93 / 19.1913.85 / 18.61
Quora22.48 / 26.6032.35 / 35.9331.49 / 35.1733.47 / 39.99
ArguAna49.53 / 97.8149.57 / 98.5950.59 / 102.3453.15 / 102.21

Values are P50/P95 in milliseconds. Against ECLIPSE-PRF, FiSA adds about 0.4 ms to the median on SciFact and 1.1 ms on Quora. Raw Dense has lower latency on both collections. Section 4.4, Figure 4, and Table A10 report latency quantiles and sequential throughput for all six collections. The benchmark evaluates the revised closed-form method.

Reviewer_ZhEuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose the Fisher-SpectralAdaptation(FISA) method, which focuses on dense retrieval and study query adaptation while avoiding retraining the foundation models. By analyzing the results of the first retrieval, the query vector is dynamically adjusted to enhance the accuracy of the second retrieval. The authors utilize experimental results to demonstrate the superiority of the method compared to the baselines, and employs ablation studies to prove the necessity of each component.

Critical strengths
  1. The research topic addressed in this paper holds significant practical implications, particularly for large models.

  2. The method proposed in this paper exhibits superior experimental results compared to most baseline methods. The authors also report results across 20 random seeds and provide component-level ablations, which helps assess the stability of the proposed approach.

Critical weaknesses
  1. However, even though there is a significant improvement in performance, the enhancement brought by the Fisher method itself is relatively limited, so the innovation of this paper is limited.

  2. Table 4 shows that the improvement brought by the Fisher-spectrum mechanism is relatively small (only 0.15 points, and not significant), while the contribution of coordinate reweighting is greater (2.99 points). Therefore, it remains unclear whether the main performance improvement in the paper truly stems from core innovations. Additionally, the method shows limited improvement on the dataset ArguAna, so its generalization ability still needs further verification.

Questions for authors
  1. Compared to the insignificant improvement of global basis, what is its advantage?

  2. Can they be compared under the same supervision conditions?

  3. What about the effect when there are fewer initial relevant documents?

  4. Can you provide a comparison of latency and throughput?

2 comments
AuthorsOpenReview ↗

The original local-versus-global comparison has a confidence interval that includes zero. With the same fitted maps and feedback, closed-form FiSA gains 0.344 nDCG@10 points over the centroid control. We also report candidate-pool sensitivity, failures when the first pass has no judged positives, and end-to-end latency.

Response to Reviewer ZhEu

Thank you for asking us to separate the effects of geometry, supervision, and candidate quality. The revision replaces the submitted neural adapter with closed-form FiSA. It keeps the frozen encoder, fixed document index, and two retrieval passes. Two fixed linear maps are fitted offline using relevance labels from NFCorpus, SciFact, and FiQA.

1. What does the local geometry add?

In the submitted five-seed ablation, local and global basis variants scored 45.935 and 45.786 nDCG@10 points, a difference of +0.149 with a 95% interval of [−0.029, 0.327]. The original global variant also changes centering. In a separate BGE-base checkpoint intervention that holds centering and learned attention fixed, the local basis gains 0.580 and 0.698 points under the two first-pass protocols. That result is specific to one checkpoint.

The revised method uses a query-local, regularized Fisher solve in candidate space. Two candidate sets can have the same weighted positive–negative centroid contrast and different within-group scatter. The Fisher solve adjusts the direction using that scatter. The implementation uses a Cholesky solve over the candidates and removes the per-query eigendecomposition. Section 3 and Appendix A give the update and its derivation.

2. How much comes from supervision?

We compare with a simpler centroid update that shares both source-fitted maps, the same candidate list, positive and negative feedback weights, and the 0.35 step. Across five encoders and six collections, closed-form FiSA scores 56.433 versus 56.089, a gain of 0.344 points with paired 95% interval [0.213, 0.477]. A separately step-tuned centroid scores 56.243. Removing the initial map or lexical feedback costs 1.639 or 1.439 points in their respective ablations. The matched Fisher increment is negative on NFCorpus and SciDocs. These comparisons quantify the direction change conditional on shared supervision and feedback (Figure 3, Table A4, Table A5).

3. What happens with fewer or weaker candidates?

The original fit excluded queries with zero judged positives in the first pass and retained them at evaluation. We now stratify closed-form evaluation queries after retrieval by the number of judged positives in Top 128. With zero judged positives, FiSA trails the same-step centroid on all six collections. For example, the differences are −1.337 on ArguAna (94 encoder–query pairs) and −1.700 on SciFact (43 pairs). The labels are used for this analysis after retrieval, and incomplete judgments can affect the strata (Table A7).

On BGE-base, Top-128 prefixes of size 32, 64, and 128 give macro scores of 57.144, 57.294, and 57.374. The step-tuned centroid scores 57.150, 57.109, and 57.082 under the same prefix protocol. We hold calibration and lexical normalization fixed, and the per-domain patterns vary (Section 4.5, Figure A1, Table A9).

4. What is the measured retrieval cost?

We benchmark the revised method and all three standalone baselines on the same Mac M5 Pro CPU with BGE-base, two threads, batch size one, and 100 queries per collection repeated three times. Timing runs from tokenization through the final Top 1000 and includes both FiSA searches. The encoder and index are resident before timing.

DatasetRaw DenseECLIPSE-PRFRocchioClosed-form FiSA
SciFact13.07 / 18.0713.46 / 18.4413.93 / 19.1913.85 / 18.61
Quora22.48 / 26.6032.35 / 35.9331.49 / 35.1733.47 / 39.99

Values are P50/P95 in milliseconds. On Quora, sequential throughput is 42.57 queries/s for Raw Dense and 29.46 for FiSA. Section 4.4, Figure 4, and Table A10 cover all six collections. These timings characterize the revised closed-form pipeline.

Thank you for your clarification, which helps me better understand your work. However, I still have concerns about innovation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.