acceptodds
Under review as a conference paper at ICLR 2027

Autoregressive Retriever: Improving Query Understanding from Item Feedback for Universal Multimodal Retrieval

Abstract

Universal multimodal retrieval typically encodes a query once and ranks independently indexed items by embedding similarity. This design supports efficient search, but leaves the query representation unchanged even when retrieved items could help clarify the information need. We introduce the AutoRegressive Retriever (ARR), a multimodal retrieval model that learns both to select informative items and to use their content to refine subsequent retrieval. ARR alternates between retrieving an item and updating the query embedding, then uses the final embedding to rank the collection. Supervised fine-tuning teaches the encoder to use feedback through stepwise contrastive supervision. Reinforcement learning treats feedback items as actions and optimizes their selection using the final reciprocal rank of a relevant item. A query-side adapter enables this optimization against a fixed item index. ARR demonstrates strong retrieval performance on both in-domain and zero-shot benchmarks, outperforming the compared baselines on average. Further analyses show that feedback improves retrieval at inference time and that training with feedback also improves the initial query embedding, before any item is observed.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Universal Multimodal Retrieval, Reinforcement Learning, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_U4WFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AutoRegressive Retriever (ARR), which repeatedly encodes the query and retrieved items to refine the query embedding without generating textual rewrites. SFT uses stepwise contrastive supervision and a degradation penalty, while RL learns feedback selection from the final reciprocal rank. Query-side LoRA keeps the item index fixed during RL. Results on M-BEIR and seven zero-shot benchmarks improve over the compared methods, with zero-shot gains already present before any feedback.

Critical strengths
  • Retrieval performance is strong, with ARR-RL-8B reaching 61.0 on M-BEIR and 79.01 on the zero-shot evaluation, exceeding the compared baselines.

  • The SFT and RL objectives are well designed. Stepwise supervision and the degradation penalty address query drift, while RL learns which feedback items improve the final ranking.

  • The ablations distinguish training-time feedback from inference-time refinement and show that simply appending retrieved content without appropriate training can hurt retrieval.

Critical weaknesses
  • My main concern is that comparisons with ELVA and TRACE use different initial models. ARR starts from the retrieval-pretrained Qwen3-VL-Embedding, but its original results are not reported. In particular, the zero-shot gains may reflect base-model differences rather than ARR's training. A same-initialization comparison with matched training budgets is needed to isolate ARR's contribution.

  • The core idea overlaps with [1], who already learn to refine query embeddings using retrieved multimodal feedback across multiple rounds. This work is not cited. Although ARR's architecture and losses differ, its added value over existing multimodal feedback refinement is not established.

  • Iterative inference adds only 0.38 points for ARR-RL-2B on M-BEIR and 0.59 points for ARR-RL-8B on zero-shot evaluation. A matched 8B single-step training control and end-to-end latency measurements are missing, leaving the training benefit and accuracy-cost trade-off unclear.

  • Results use one training seed, despite some differences being only 0.10 points. These small gains need an uncertainty analysis.

[1] Bulat Khaertdinov, Mirela Popa, and Nava Tintarev. A Little More Like This: Text-to-Image Retrieval with Vision-Language Models Using Relevance Feedback. WACV 2026.

Questions for authors
  • Can you report the original Qwen3-VL-Embedding results and compare ARR with ELVA or TRACE from the same initialization and training budget, including an 8B single-step control? Alternatively, can you train both ARR and ELVA from ELVA's embedding-pretrained checkpoint under matched training budgets?

  • How does ARR differ from the multimodal feedback refinement approach in [1]?

  • How does ARR compare with feeding the initial top-k retrieved items together and refining the query embedding only once, using the same backbone and matched training and input budgets?

  • What are the end-to-end latency and throughput for one, two, and four query states?

Reviewer_hmZVOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Auto Regressive Retriever (ARR), a multimodal retrieval model that alternates between retrieving an item and updating the query embedding. ARR has two stages: Supervised fine-tuning (SFT) first teaches the encoder to interpret feedback using stepwise contrastive supervision on offline trajectories. Reinforcement learning (RL) then learn feedback selection from trajectories sampled by the current policy, using final retrieval quality as the reward. ARR demonstrates strong retrieval performance on both in-domain and zero-shot benchmarks, outperforming the compared baselines on average.

Critical strengths
  • The experiments suggest that models trained to interpret retrieved feedback can repeatedly leverage feedback without suffering from the severe degradation observed in the N=1 setting, which is often a concern in pseudo-relevance feedback approaches. Further analyses show that feedback improves retrieval at inference time and that training with feedback also improves the initial query embedding.
  • The paper demonstrates that learning to select informative feedback items through RL further improves retrieval performance beyond what can be achieved by SFT-based feedback utilization alone
  • The paper effectively validates its claims through a sufficiently large and diverse set of benchmark datasets and comprehensive ablation studies.
Critical weaknesses

None

Questions for authors
  • When running Iter-1 with SFT only, which retrieved item is used as the feedback item?
  • It would also be helpful to provide an analysis of the differences between the feedback items selected by SFT and RL in practice. Understanding what kinds of items RL learns to select, compared to SFT, would give more insight into the characteristics of the proposed retrieval policy and the source of the observed performance gains.
Reviewer_6dRzOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ARR, a multimodal retriever that iteratively updates its query embedding using retrieved items as feedback while keeping item embeddings independently indexed. SFT teaches the model to use feedback through contrastive supervision, and RL trains item selection based on the final retrieval ranking. Experiments with 2B and 8B models on M-BEIR and zero-shot benchmarks report improvements over the compared baselines.

Critical strengths

The idea of learning which items to retrieve based on their usefulness for subsequent retrieval is creative.

Critical weaknesses
  1. Latency is not evaluated. The approach adds feedback rounds, yet reports no end-to-end latency or throughput measurements. Latency is critical in retrieval, and I cannot judge whether the gains justify this additional cost.
  2. The baseline selection needs clarification. Why were these retrievers chosen? Do they represent the current state of the art on these benchmarks, or serve another purpose? Results for the original Qwen3-VL-Embedding checkpoint would help isolate ARR’s contribution. The single-step training control is only reported for 2B on M-BEIR; extending it to 8B and the zero-shot benchmarks would make the gains easier to interpret.
  3. The experimental gains seem very small in many cases, with no confidence intervals to establish statistical significance. For example, feedback improves ARR-RL-2B from 56.36 to 56.74 in Table 2 - just 0.38 points.
  4. The paper lacks comparison to top-k document reranking, which seems like a natural alternative to this idea.
Questions for authors
  1. The coverage of pseudo-relevance feedback literature as related work is incomplete (Yu et al., 2021; Sung et al., 2023; Gangi Reddy et al., 2025; Uzan et al., 2026).
  2. Can proper latency comparisons be added?
  3. Can confidence intervals and a seed comparison be added?

Comments on the paper

1 comment
Area_Chair_9KFfOpenReview ↗

Please read the rebuttal and engage in discussion

Dear Reviewers,

The authors have posted their rebuttal. Please read it and engage in discussion, in particular:

  • Whether each of your questions and concerns has been addressed.
  • Whether your rating and recommendation have changed, and why.
  • Any remaining concerns that must be settled before the meta-review.

Thanks for your time and contribution.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.