acceptodds
Under review as a conference paper at ICLR 2027

TRACE: Task-Adaptive Reasoning and Representation Learning for Universal Multimodal Retrieval

13 views

Abstract

Universal multimodal retrieval must map both explicit matches and unresolved compositional intents into a shared embedding space. Intermediate reasoning can clarify a query's target semantics, but generating it for every input adds latency and may weaken retrieval. We introduce TRACE, a framework that learns retrieval representations through selective query-side reasoning. Its first autoregressive prediction selects immediate embedding or a retrieval-oriented reasoning sequence; both routes produce a vector from the hidden state predicting the same embedding marker. Joint generation and contrastive training couple intent resolution to representation learning, while directly encoded candidates remain indexable offline. We construct M-BEIR-CoT with 518K direct examples and 575K filtered reasoning examples to supervise both behaviors. TRACE achieves a 58.8 average score on M-BEIR, improving a same-backbone LamRA-Ret reproduction by 2.1 points, and improves over LamRA-Ret in 11 of 13 distribution-shifted settings. On separate MSCOCO and CIRR controls, it attains higher recall than independently trained direct-only and always-reasoning policies, with approximately the query throughput of always reasoning. CIRR interventions and ablations further support reasoning-enabled inference, joint representation learning, and query-only generation. Together, these results demonstrate selective reasoning within a shared, index-compatible multimodal retriever.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Universal multimodal retrieval, Representation learning, Adaptive reasoning, Multimodal embeddings, Contrastive learning, Efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_32MtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRACE combines selective query-side reasoning with dense multimodal retrieval. The first generated token chooses between immediate embedding and a retrieval-oriented reasoning sequence, and both routes use the hidden state predicting the same embedding marker. Candidates are encoded directly, preserving offline indexing. The authors construct mixed direct/reasoning supervision and jointly optimize generation and contrastive losses. TRACE reaches 58.8 on M-BEIR versus 56.7 for a backbone-matched LamRA-Ret reproduction, with additional transfer experiments and controlled studies on MSCOCO and CIRR.

Critical strengths
  1. The writing is generally clear, and the design is clean and practical. The first-token decision avoids an additional routing network and is simple yet effective.

  2. The main retrieval gains are consistent: TRACE improves over the backbone-matched LamRA-Ret reproduction in all 16 settings in Table 1. The experiments are also comprehensive, with transfer evaluations covering composition, dialogue, interleaved context, and negative constraints.

Critical weaknesses
  1. The backbone-matched LamRA-Ret comparison in the main table demonstrates the effectiveness of the overall method. However, it does not sufficiently establish the additional benefits of the proposed interface over existing reasoning-based or gated retrievers. I would therefore like to see comparisons with more closely related reasoning-based or adaptive retrieval methods, such as MMEmb-R1 and Think When Needed, using the same backbone, supervision data, and training budget. Such comparisons would help clarify the value of first-token routing and the shared representation interface, which together constitute the paper's core contribution.

  2. The supervision for both when to reason and how to reason comes entirely from the teacher model, GPT-4.1. This raises the concern that the method may largely amount to distilling the teacher model's behavior. It also substantially limits the methodological novelty. Moreover, the current experiments do not establish that the routing mechanism selects queries that actually need reasoning. The paper reports that approximately 96% of MSCOCO queries are encoded directly, while approximately 62% of CIRR queries follow the reasoning path. This shows that the model adopts different strategies across tasks, but does not directly demonstrate that the queries selected for reasoning are those that benefit most from it. I suggest evaluating both paths on the same queries and comparing the learned routing policy with random routing and fixed task-based routing under the same reasoning budget. This is my main concern regarding the paper's central claim of adaptivity.

For me, this is borderline, with a slight negative lean. The authors address a valuable problem, and the method appears to offer some benefits. However, several key claims lack experimental support, and the methodological novelty is limited. Overall, I find the academic contribution somewhat insufficient.

Questions for authors

See wkness

Reviewer_FQWaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TRACE combines query-side adaptive reasoning with multimodal retrieval representation learning. Based on its first generated token, the model either emits the embedding marker immediately or first generates a retrieval-oriented reasoning/answer sequence. To train TRACE, the authors construct M-BEIR-CoT, a supervised training corpus in which GPT-4.1 assigns each query to the direct or reasoning route and generates the corresponding reasoning content. TRACE achieves an average score of 58.8 on the M-BEIR benchmark, 2.1 points higher than a same-backbone LamRA-Ret reproduction, and outperforms LamRA-Ret in 11 of 13 distribution-shifted settings. Controlled comparisons on MSCOCO and CIRR suggest that adaptive reasoning improves retrieval accuracy while retaining higher query throughput than an always-reasoning model.

Critical strengths
  • The paper highlights a meaningful issue for universal multimodal embedding models: reasoning may incur extra inference latency, and unnecessary or unsuitable reasoning may also reduce retrieval accuracy. TRACE addresses this issue by allowing the model to choose, for each query, between direct embedding generation and reasoning, offering a principled way to balance retrieval quality and inference efficiency.
  • The paper reports a fairly comprehensive set of experiments. The results suggest that TRACE’s selective-reasoning policy allocates query-side computation effectively, improving retrieval performance on M-BEIR while maintaining higher query throughput than the separately trained always-reasoning controls on MSCOCO and CIRR.
Critical weaknesses
  • The central claim of TRACE is that it can decide when a query should undergo reasoning and when it should be encoded directly as an embedding. However, this selective-reasoning ability is supervised using route labels generated by GPT-4.1, and the paper does not sufficiently explain or validate the basis for these decisions. Three core aspects of the proposed mechanism therefore remain unclear:

    1. Basis for route assignment. The paper does not describe the teacher’s decision process in a sufficiently transparent and reproducible manner, nor does it analyze the criteria or decision boundary for determining whether a query requires reasoning. Both empirical analysis and theoretical discussion are lacking.
    2. Behavior of the trained router. The paper does not systematically show which query features lead TRACE to select reasoning or direct encoding, how these choices vary across task types, whether the decisions are appropriate, or whether the policy remains robust when query characteristics and task settings change. In particular, Table 3 reports remarkably similar relative QPS changes for MSCOCO and CIRR. Although QPS is affected by multiple factors and is not a direct measure of route frequency, this similarity raises the concern that TRACE may have learned an approximately fixed reasoning rate rather than making meaningfully query-dependent decisions. The paper should report detailed route frequencies and associated retrieval and efficiency statistics by dataset and query type to demonstrate that the model has learned a useful adaptive policy.
    3. Effect of route-label supervision. The aggregate results do not show how sensitive TRACE is to the route labels used for training, whether the current labeling strategy is necessary, or how alternative, noisy, or systematically perturbed labels would affect retrieval quality and efficiency. Taken together, these gaps mean that the paper reports the method and its overall results without sufficiently establishing the effectiveness, necessity, and robustness of its central routing mechanism. This substantially limits the persuasiveness of the paper’s contribution.
  • The paper has recurring issues with writing quality, particularly imprecise terminology and overly compressed phrasing. These issues make it harder for readers to understand the proposed method accurately and to follow the paper smoothly. In particular:

    1. Undefined and inconsistently specified terminology. Several central terms are not clearly defined or consistently delimited. The paper uses “reasoning sequence,” “retrieval-oriented sequence,” and “reasoning + answer” for generated outputs without establishing whether they refer to the same object or distinct components. It also introduces “retrieval relation” as (u) for the query and candidate instructions without explaining what this relation means. These gaps make the concepts and their relationships difficult to follow.
    2. Imprecise and overly compressed wording. The paper sometimes uses informal shorthand or leaves referents and relationships implicit, making it difficult to determine exactly what a phrase means. For example, “MSCOCO and CIRR policy controls are trained separately with matched backbones, pairs, schedules, and update counts” leaves “policy controls” undefined, does not state which models are matched, and uses the underspecified term “pairs.” The authors should define such shorthand and state the relevant entities and relationships explicitly throughout the paper.
Questions for authors
  1. Could you provide a transparent and reproducible description of the criteria and procedure GPT-4.1 uses to generate the route labels? How do you verify that the labels identify queries for which reasoning improves retrieval, rather than merely reflecting the teacher model’s judgments or dataset conventions? Could a simple, interpretable baseline help assess this—for example, testing whether providing additional reasoning context to an existing embedding model such as LamRA-Ret improves retrieval accuracy?
  2. What determines whether the trained TRACE model selects the reasoning route or directly emits an embedding? Could you analyze the query features associated with each decision, report route frequencies and the corresponding retrieval and latency outcomes by task type or dataset, and test whether these behaviors remain stable as query complexity, instructions, or modality change?
  3. How sensitive are TRACE’s retrieval quality and efficiency to the route labels used for training? If models with different capabilities—both weaker and stronger than GPT-4.1—were used to generate the route labels and construct the training data, how would the resulting TRACE models compare in retrieval accuracy and inference efficiency?
Reviewer_pcbBOpenReview ↗
Rating
2
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRACE is a multimodal retrieval framework that allows the model to choose between direct encoding and generating additional text before producing a retrieval embedding. The paper also presents M-BEIR-CoT to support both behaviors during training. While the reported results are encouraging, the experiments leave some questions about the source and reliability of the improvements.

Critical strengths

1.The paper explores a useful direction for multimodal retrieval by adapting the amount of reasoning to the query rather than applying the same strategy to every input.

2.The evaluation covers a range of retrieval tasks, and the ablation studies examine several important design choices, including the reasoning supervision and the selection of the embedding representation.

Critical weaknesses

1.The performance gains cannot be fully explained by the current experiments. TRACE combines synthetic reasoning traces, adaptive routing, and joint training of generation and retrieval. It remains unclear how much each component contributes to the final results.

2.The efficiency results depend on the reference method. Although TRACE reduces the cost of always generating reasoning, it is slower than direct encoding on the overall M-BEIR workload. The paper would benefit from a clearer discussion of when the accuracy improvement justifies the extra inference cost.

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.