acceptodds
Withdrawn from ICLR 2027

Why Only the Final Thought? Hierarchical Aggregation for Multimodal Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) have emerged as powerful backbones for learning embeddings for multimodal retrieval. Recent methods incorporate chain-of-thought (CoT) reasoning into embedding generation and further employ adaptive routing mechanisms to selectively invoke reasoning and control its computational cost. However, different queries may benefit from embeddings formed with different reasoning-token budgets and at different model depths, whereas existing methods typically rely on a single final embedding. To address this limitation, we introduce a dual-aggregator framework that exploits these embeddings, adaptively combining them for each query and aggregating their matching evidence to rank candidates. Experiments on MMEB-V2 validate the benefits of integrating information across reasoning-token budgets and model depths, with our framework achieving strong overall retrieval performance with less than 1% additional parameters over the backbone. Further analytical experiments demonstrate that our framework adapts its aggregation to individual queries, effectively integrating information across budgets and depths to improve retrieval performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal large language models, multimodal retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RYpFOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DUET, a multimodal retrieval framework that uses representations from different continuous-token budgets and decoder depths. It first trains multiple retrieval exits, then combines representations through a Query Aggregator and integrates matching evidence through a Candidate Aggregator for reranking. The method achieves good performance on MMEB-V2.

Critical strengths
  1. The paper brings the established idea of aggregating complementary representations across budgets and depths to MLLM embeddings, with good empirical results.
  2. The method achieves promising retrieval performance without generating explicit textual CoT at inference, avoiding the overhead of autoregressive rationale generation. However, it still requires multiple decoder forward passes, so its efficiency advantage depends on the comparison setting.
  3. The appendix provides detailed method definitions, exit-wise analyses, and runtime measurements. I appreciate the authors' effort in documenting these details.
Critical weaknesses
  1. The method is complex, but targeted ablations establishing the necessity of its components are missing. This is my main concern.

    • Training objectives in Section 3.1: LexitL_{\mathrm{exit}} includes final-exit supervision, all-exit supervision, cross-exit consistency, and CoT distillation, but their individual contributions are not evaluated. For example, whether the CoT teacher provides useful gains, and how the loss coefficients were chosen, remain insufficiently supported or explained.
    • The auxiliary loss in Section 3.2: No results are reported without LQ,auxL_{Q,\mathrm{aux}}. The connection between its label construction and the retrieval objective needs further explanation, as does the choice of its coefficient, 0.25.
    • Weight-prediction features in Section 3.2: Query metadata, budget/depth identifiers, and cross-depth comparison features are introduced together without isolating their contributions. Given the substantial improvement attributed to the Query Aggregator in Table 2, finer-grained comparisons are needed to identify which designs actually drive the gains.
    • Matching and reranking in Section 3.3: The combined matching vector incorporates semantic matching, retrieval statistics, and source identifiers, followed by two attention modules. Existing experiments primarily remove the entire module or replace it with non-learned fusion, leaving the necessity of these individual components unclear.
  2. The additional value of shallow and low-budget representations is not sufficiently explained. Shallow raw representations receive relatively little attention in Figure 3, and many fixed-depth budget comparisons in Figure 6 show small performance differences. The budget-diversity ablations improve the overall score by 0.2–0.3 points. These observations do not establish that such representations are unhelpful, but more direct evidence is needed to explain what complementary information the complex aggregation scheme exploits and whether a reduced representation set could retain most of the gains.

  3. The main table should group methods by comparable backbones and similar model sizes. This would make the relative improvements easier to assess and provide a fairer, clearer presentation of differences in model scale and retrieval pipeline.

  4. The main-text presentation needs improvement. The method contains many components, but their descriptions are often brief. For example, L211–L218 introduce the components of xk,dj(q)x_{k,d_j}(q) without sufficiently clear definitions or motivation, requiring frequent consultation of the appendix. For a method of this complexity, the main text should explain the principal inputs, module connections, and design choices more clearly.

Questions for authors

Overall, this paper proposes an MLLM embedding method that aggregates representations across different decoder depths and computational budgets (although the budget difference amounts to only a few learnable tokens, whose incremental computational cost is minimal relative to processing the hundreds or thousands of prefill tokens).

My main concern is not whether the complete system improves performance, but whether these improvements require the proposed complexity, and what each loss, input feature, and submodule contributes.

Sections 3.2 and 3.3 introduce numerous submodules and input features, while both stages produce substantial improvements. This makes component-wise ablations particularly important, at a finer granularity than the comparisons in Table 2. The design gives me the impression of progressively stacking building blocks: each additional module, feature, and training objective increases the burden of establishing its necessity. The current experiments do not clearly identify which components drive the gains and which could be omitted.

Although I personally dislike reviewers requesting mechanical, exhaustive ablations, the complexity of this particular design creates a substantial need for more targeted validation. If the authors can clearly explain why each module and its inputs are necessary, with convincing justification where appropriate, I would be able to better understand these design choices.

At present, I remain concerned that the design and its gains may primarily reflect the accumulation of components rather than a clearly established new mechanism. I therefore consider the novelty limited and retain a negative assessment. That said, if the other reviewers and the AC all maintain a positive assessment, I would not object to acceptance.

Reviewer_yxpAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DUET, a hierarchical aggregation framework for multimodal retrieval.

Existing reasoning-enhanced embedding methods rely on a single representation produced at the final layer under a fixed reasoning budget. DUET extracts multi-exit query representations across different continuous reasoning budgets and decoder depths. It uses a Query Aggregator to learn query-dependent weights across reasoning budgets for each depth, and a Candidate Aggregator to combine multi-view evidence and model candidate interactions for final ranking. Evaluated on MMEB-V2 across 78 tasks, DUET-4B and DUET-8B achieve good performance.

Critical strengths
  1. The parameter overhead remains minimal relative to the backbone. The continuous reasoning tokens, layer adapters, and dual aggregators add 25.15M parameters to the 4B model and 40.03M to the 8B model, both below 1% of the respective backbone size.

  2. The paper provides extensive ablations validating the utility of budget and depth diversity.

  3. The motivation behind exploring multi-exit representations is well-articulated.

Critical weaknesses
  1. The overall performance gain is marginal and lacks variance reporting. DUET-8B improves over TWN-8B by only 0.5 points overall (69.2 v.s. 68.7) and actually underperforms TWN-8B on both Image (71.2 v.s. 73.4) and VisDoc (76.8 v.s. 77.0) tasks, with no multi-seed runs or standard deviations provided to confirm whether this gain is statistically significant.

  2. The evaluation mixes embedding retrieval with candidate reranking against pure embedding baselines. The full DUET pipeline uses 21 query representations to retrieve candidates into a shared pool and then runs a Candidate Aggregator for listwise reranking, whereas baselines like VLM2Vec, GME, and TWN perform standard single-vector retrieval.

  3. The multi-exit extraction incurs considerable inference latency. As reported in Table 6, full evaluation takes 2.0x to 3.2x longer wall-clock time compared to the direct backbone, because DUET runs the decoder multiple times for different reasoning budgets to extract the representations.

  4. A large performance gap remains between the dual aggregator and the exit-selection oracle. Table 1 shows that DUET-8B achieves 69.2 while the oracle reaches 76.6, indicating that the aggregation mechanism leaves a substantial 7.4-point margin uncaptured despite aggregating 21 representations.

Questions for authors

Please kindly refer to the weaknesses.

Reviewer_weqfOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper challenges the common practice of using only the final representation in MLLM-based multimodal embeddings and proposes DUET, which jointly leverages representations obtained at multiple reasoning budgets and layer depths.

  • Multi-exit representation learning: The authors freeze a Qwen3-VL backbone fine-tuned with contrastive learning, then append k learnable continuous latent reasoning tokens and a readout token. Representations are extracted at four depths (18, 24, 30, and 36), yielding 16 exits. The training process incorporates selective distillation from a CoT-conditioned teacher and a cross-exit consistency loss.

  • Query Aggregator: At each layer depth, representations from different reasoning budgets are combined using query-dependent weights, producing four aggregated representations.

  • Candidate Aggregator: The top-5 candidates retrieved by each of the 21 query representations are merged into a candidate pool. Attention summarizes the matching evidence from each representation, while self-attention models interactions between candidates to produce the final ranking.

On MMEB-V2, which comprises 78 tasks, DUET-8B achieves an overall score of 69.2, outperforming the compared methods. The authors also claim that the additional parameters account for less than 1% of the backbone's parameter count.

Critical strengths
  1. Comprehensive comparative experiments and ablation studies. The authors systematically investigate the contributions of individual design components through parameter-matched comparisons with alternative aggregation methods, including Router, ARM, and simple averaging (Table 8); ablations removing each aggregator and varying budget/depth diversity (Tables 2 and 7); and validation across both 4B and 8B model scales.

  2. Strong performance on video tasks and insightful analysis. DUET outperforms the strongest baseline (53.0) on the overall video score, achieving 53.5 with the 4B model and 54.8 with the 8B model, with substantial improvements in video QA, retrieval, and moment retrieval. The exit-wise performance analysis (Figure 6) and the oracle results for exit selection (76.6 for the 8B model) provide meaningful evidence that intermediate representations contain substantial complementary information.

I appreciate the authors' thorough experimental investigation.

Critical weaknesses

The SOTA claim relies heavily on a learned reranker, making the comparisons potentially unfair.

The Candidate Aggregator is effectively a listwise reranker: it reranks candidates within a retrieved pool of up to approximately 110 items by applying self-attention across candidates. However, most competing methods rely on a single embedding representation.

Without this reranking component, DUET fails to outperform several baselines:

  • Removing the Candidate Aggregator reduces the overall 8B score to 64.6 (Table 2).
  • Endpoint Averaging achieves 66.2, Router 66.7, and ARM 67.1 (Table 8).
  • In contrast, TWN-8B achieves 68.7, TTEs-7B 68.6, and Embed-RL-4B 68.1.

These results suggest that adding a reranking stage, rather than exploiting representation diversity across reasoning budgets and layer depths, may be the primary driver of the claimed state-of-the-art performance.

Questions for authors
  1. What is the MMEB-V2 score of the contrastively trained backbone without any additional DUET components (Plain)? Please include this result in Table 1.
  2. Could you also compare DUET against existing multimodal rerankers using the same candidate pool?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.