Q-REACT: Query Residual Embedding Adaptation through Conditional Transformations for Visual Document Retrieval

Visual document retrieval (VDR) relies on an offline constructed page index, yet improving retrieval against this fixed index often requires costly model training or access to encoder parameters, which may be unavailable for proprietary APIs. Multimodal rerankers provide high-quality relevance signals, but under limited budgets they assess only a small subset of queries and candidate pages. Their gains therefore remain local to candidate reordering, leaving useful supervision unused for improving the retrieval system as a whole. We propose Q-REACT (**Q**uery **R**esidual **E**mbedding **A**daptation through **C**onditional **T**ransformations), a test-time optimization method that reuses reranker rewards to improve retrieval over the full index with limited feedback. Q-REACT combines query-dependent low-rank residual adaptation with document-context scoring while keeping the encoders and page index fixed. It distills local reranker preferences using cross-entropy with a student distribution normalized over the complete task-specific index, allowing unscored pages to participate as competitors through cached embeddings without additional reward calls. Experiments across eight ViDoRe V3 tasks and five open-weight and proprietary backbones show higher average retrieval quality than direct reranking and TTT-Embed at representative sparse and full-coverage feedback budgets. Our findings show that Q-REACT makes more effective use of limited reranker feedback, improving retrieval across the query collection without retraining encoders or rebuilding the page index.