acceptodds
Under review as a conference paper at ICLR 2027

Q-REACT: Query Residual Embedding Adaptation through Conditional Transformations for Visual Document Retrieval

Abstract

Visual document retrieval (VDR) relies on an offline constructed page index, yet improving retrieval against this fixed index often requires costly model training or access to encoder parameters, which may be unavailable for proprietary APIs. Multimodal rerankers provide high-quality relevance signals, but under limited budgets they assess only a small subset of queries and candidate pages. Their gains therefore remain local to candidate reordering, leaving useful supervision unused for improving the retrieval system as a whole. We propose Q-REACT (**Q**uery **R**esidual **E**mbedding **A**daptation through **C**onditional **T**ransformations), a test-time optimization method that reuses reranker rewards to improve retrieval over the full index with limited feedback. Q-REACT combines query-dependent low-rank residual adaptation with document-context scoring while keeping the encoders and page index fixed. It distills local reranker preferences using cross-entropy with a student distribution normalized over the complete task-specific index, allowing unscored pages to participate as competitors through cached embeddings without additional reward calls. Experiments across eight ViDoRe V3 tasks and five open-weight and proprietary backbones show higher average retrieval quality than direct reranking and TTT-Embed at representative sparse and full-coverage feedback budgets. Our findings show that Q-REACT makes more effective use of limited reranker feedback, improving retrieval across the query collection without retraining encoders or rebuilding the page index.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Visual document retrieval, Test Time, MultimodalEmbedding, Reranker, Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Gk69OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Q-REACT, a test-time query embedding adaptation method for visual document retrieval. The approach learns query-dependent low-rank residual transformations from limited reranker feedback while keeping the retrieval encoders and document index frozen. It further incorporates document-context scoring and full-index distillation to improve retrieval performance. Experiments on eight ViDoRe V3 tasks with five multimodal retrieval backbones demonstrate improvements over direct reranking and TTT-Embed across different feedback budgets.

Critical strengths

1.The paper addresses an important problem: improving retrieval performance with limited reranker feedback without modifying pretrained encoders or rebuilding document indexes. This setting is particularly relevant to proprietary embedding APIs and large-scale retrieval systems.

2.The experiments cover eight ViDoRe V3 tasks, five retrieval backbones, and multiple feedback budgets and reward models. Q-REACT achieves substantial improvements over direct reranking and TTT-Embed.

Critical weaknesses

1.The proposed query-dependent low-rank residual, Δ(x)=BAx/√r, is essentially a shared low-rank linear transformation with a fixed transformation matrix after adaptation. Prior methods have explored more explicitly conditioned retrieval transformations. For example, QuARI (NeurIPS 2025) employs a hypernetwork to generate query-specific low-rank projections, while CAST (CVPR 2026) introduces context-conditioned latent space transformations for interactive retrieval. Although their tasks and training settings differ from Q-REACT, these studies are closely related to its central transformation mechanism. The paper does not adequately discuss these connections or compare against simpler, parameter-matched transformation alternatives, making the novelty of the proposed representation adaptation less convincing.

Related papers:

QuARI(NIPS 2025)https://papers.neurips.cc/paper_files/paper/2025/file/5ae6634cb92414dc34d24e863e8d0809-Paper-Conference.pdf

CAST(CVPR 2026)https://openaccess.thecvf.com/content/CVPR2026/papers/Lin_CAST_Context-Aware_Dynamic_Latent_Space_Transformation_for_Interactive_Text-to-Image_Retrieval_CVPR_2026_paper.pdf

2.The authors motivate the proposed low-rank transformation by arguing that a fixed residual vector cannot adequately accommodate heterogeneous query intents. However, this primarily motivates a more expressive parameterization rather than explaining why transforming query embeddings is an effective solution to the underlying retrieval problem. The paper does not clearly identify specific semantic or geometric misalignments in the frozen query–document embedding space, nor does it explain why the required corrections should be captured by a shared low-rank linear mapping. Although the ablations demonstrate empirical improvements over fixed residuals, they provide limited insight into the underlying mechanism.

3.The paper argues that full-index normalization improves retrieval by introducing competition from unscored pages. However, Table 3 shows that, with document context enabled, full-index normalization improves the Task setting by 1.14 points but reduces the Global setting by 0.38 points compared with Top-10 normalization. Moreover, the full-index objective encourages the rewarded Top-K candidates to receive greater probability mass, implicitly treating unscored pages as competitors despite their potentially containing relevant documents. This introduces a possible false-negative issue. The paper lacks sufficient empirical analysis of this assumption, particularly across sparse feedback budgets.

Questions for authors

1.How does the proposed low-rank query transformation fundamentally differ from existing query-adaptive retrieval methods such as QuARI and CAST? Why are these closely related studies not discussed, and could the authors provide comparisons with parameter-matched query transformation baselines?

2.Beyond increased representational capacity compared with a fixed residual vector, what specific limitations of the frozen query–document embedding space motivate the proposed transformation? Could the authors provide empirical evidence that the learned mapping addresses particular semantic or geometric misalignments?

3.Table 3 shows that full-index normalization underperforms Top-10 normalization in the Global setting. Why is full-index normalization adopted for both sharing modes? How does its effectiveness change under sparse feedback, especially when initially unscored pages may be relevant?

Reviewer_x61kOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Q-REACT, a test-time method for visual document retrieval that learns adaptive query transformations from limited reranker feedback, with encoders and document embeddings frozen. Q-REACT integrates low-rank residuals into query embeddings for better task alignment, and trains the residual matrices via distillation. The teacher provides feedback based on only a small set of candidates, while the student learns from the feedback based on global retrieval. Experimental results demonstrate the consistent performance improvement of Q-REACT across multiple tasks and models.

Critical strengths
  • Clear problem formulation: Q-REACT targets the problem of efficiency and limited teacher feedback, which is a common practical scenario in the field.
  • Strong performance: Q-REACT improves retrieval performance across multiple models and task settings. The ablation studies are also detailed and comprehensive.
Critical weaknesses
  • Effectiveness of training under limited feedback: the method neglects specific cases where true positives (or high-score candidates) do not appear in the top-k retrieval results. In these cases, the scores of the true positives may be suppressed.
  • Limited test coverage: the main experiments are conducted only on ViDoRe V3 with only one fixed seed.
Questions for authors
  • Can the authors provide additional experimental results (e.g., different seeds, datasets, or model families) to further support the claims of Q-REACT?
  • Can the authors provide additional details about hyperparameters and training details? How are the settings chosen? Why does Q-REACT adopt the stated settings?
Reviewer_6HhWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Q-REACT, a lightweight test-time adaptation method for visual document retrieval (VDR). The method keeps the embedding encoders and document index frozen, while using limited reranker feedback to train a reusable low-rank query adapter.

The main idea is to transfer the benefits of reranker feedback from a small subset of queries to the overall retrieval system, with limited additional computational cost.

The method combines three components:

  1. Query-dependent low-rank residual adaptation to adjust query embeddings.
  2. Document-context-aware scoring to incorporate document-level relevance information.
  3. Full-index distillation to learn from reranker feedback while considering competing pages beyond the rewarded candidates.

The paper evaluates Q-REACT on eight ViDoRe V3 tasks across five embedding backbones, showing improvements under both sparse and full feedback budgets.

Critical strengths
  1. Practical and meaningful problem. The paper addresses an important system optimization problem: improving retrieval quality without modifying the embedding model or rebuilding the document index. Reusing limited reranker feedback through a lightweight query transformation is a useful approach, particularly for deployment scenarios involving frozen or proprietary embedding models.

  2. Solid experimental results. The method demonstrates clear improvements under both sparse and full feedback settings, with positive gains across all five evaluated backbones. The experiments include comparisons with relevant baselines, feedback-budget scaling, ablation studies, generalization evaluations, and inference latency measurements. Overall, the empirical results provide reasonably strong support for the effectiveness of the method.

Critical weaknesses
  1. The methodological novelty is relatively incremental.

    • The method mainly combines existing ideas, including low-rank residual adaptation, document-context scoring, and reranker-guided distillation.
    • Although the combination is effective, the technical contribution is more of an incremental improvement over existing test-time query adaptation approaches.
  2. Generalization improvements are relatively limited.

    • The method shows positive results on held-out queries and leave-one-task-out evaluation, but the improvements are smaller than those observed in the main full-feedback experiments.
    • It remains unclear how well the learned query transformations generalize to different document collections, retrieval domains, or changing indexes.
Questions for authors
  1. Generalization: Can the authors provide further evaluation on unseen document collections or different retrieval domains? The current held-out-query and leave-one-task-out results are positive, but it would be useful to understand whether the method remains effective under larger distribution shifts.

  2. Contribution of full-index normalization: The ablation results show that full-index normalization improves Task adaptation but does not consistently outperform Top-10 normalization in Global mode. Could the authors explain this difference and clarify under which conditions full-index normalization is beneficial?

  3. Contribution of document context: Since document-context scoring already improves retrieval without reranker feedback, could the authors further clarify how much of the improvement under sparse feedback comes from the learned query transformation versus the document-level prior?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.