Relevance Is Not Variance: Local Fisher-Spectral Adaptation for Dense Retrieval

Retrieval-augmented generation (RAG) relies on relevant external evidence to ground language model responses. Retrieving this evidence remains challenging when a fixed embedding space must serve queries with different information needs. Directions that capture substantial variation in that space may offer little help in distinguishing relevant evidence from distractors. We study how relevance feedback can guide query adaptation while keeping the encoder and document index unchanged. We propose Fisher-Spectral Adaptation (FISA), a supervised adapter that uses the initial retrieval results to identify locally discriminative directions and refine the query for a second search. Experiments with five frozen embedding models on four BEIR tasks show a 3.63-point gain in macro nDCG@10 (8.58% relative) over the strongest deterministic baseline, with improvements in 16 of 20 encoder–dataset comparisons. The gains are stable across 20 training seeds, while evaluation on an unseen domain reveals limits to transfer. These results demonstrate the potential of adapting retrieval geometry to query-specific evidence.