- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DUET, a multimodal retrieval framework that uses representations from different continuous-token budgets and decoder depths. It first trains multiple retrieval exits, then combines representations through a Query Aggregator and integrates matching evidence through a Candidate Aggregator for reranking. The method achieves good performance on MMEB-V2.
Critical strengths
- The paper brings the established idea of aggregating complementary representations across budgets and depths to MLLM embeddings, with good empirical results.
- The method achieves promising retrieval performance without generating explicit textual CoT at inference, avoiding the overhead of autoregressive rationale generation. However, it still requires multiple decoder forward passes, so its efficiency advantage depends on the comparison setting.
- The appendix provides detailed method definitions, exit-wise analyses, and runtime measurements. I appreciate the authors' effort in documenting these details.
Critical weaknesses
-
The method is complex, but targeted ablations establishing the necessity of its components are missing. This is my main concern.
- Training objectives in Section 3.1: includes final-exit supervision, all-exit supervision, cross-exit consistency, and CoT distillation, but their individual contributions are not evaluated. For example, whether the CoT teacher provides useful gains, and how the loss coefficients were chosen, remain insufficiently supported or explained.
- The auxiliary loss in Section 3.2: No results are reported without . The connection between its label construction and the retrieval objective needs further explanation, as does the choice of its coefficient, 0.25.
- Weight-prediction features in Section 3.2: Query metadata, budget/depth identifiers, and cross-depth comparison features are introduced together without isolating their contributions. Given the substantial improvement attributed to the Query Aggregator in Table 2, finer-grained comparisons are needed to identify which designs actually drive the gains.
- Matching and reranking in Section 3.3: The combined matching vector incorporates semantic matching, retrieval statistics, and source identifiers, followed by two attention modules. Existing experiments primarily remove the entire module or replace it with non-learned fusion, leaving the necessity of these individual components unclear.
-
The additional value of shallow and low-budget representations is not sufficiently explained. Shallow raw representations receive relatively little attention in Figure 3, and many fixed-depth budget comparisons in Figure 6 show small performance differences. The budget-diversity ablations improve the overall score by 0.2–0.3 points. These observations do not establish that such representations are unhelpful, but more direct evidence is needed to explain what complementary information the complex aggregation scheme exploits and whether a reduced representation set could retain most of the gains.
-
The main table should group methods by comparable backbones and similar model sizes. This would make the relative improvements easier to assess and provide a fairer, clearer presentation of differences in model scale and retrieval pipeline.
-
The main-text presentation needs improvement. The method contains many components, but their descriptions are often brief. For example, L211–L218 introduce the components of without sufficiently clear definitions or motivation, requiring frequent consultation of the appendix. For a method of this complexity, the main text should explain the principal inputs, module connections, and design choices more clearly.
Questions for authors
Overall, this paper proposes an MLLM embedding method that aggregates representations across different decoder depths and computational budgets (although the budget difference amounts to only a few learnable tokens, whose incremental computational cost is minimal relative to processing the hundreds or thousands of prefill tokens).
My main concern is not whether the complete system improves performance, but whether these improvements require the proposed complexity, and what each loss, input feature, and submodule contributes.
Sections 3.2 and 3.3 introduce numerous submodules and input features, while both stages produce substantial improvements. This makes component-wise ablations particularly important, at a finer granularity than the comparisons in Table 2. The design gives me the impression of progressively stacking building blocks: each additional module, feature, and training objective increases the burden of establishing its necessity. The current experiments do not clearly identify which components drive the gains and which could be omitted.
Although I personally dislike reviewers requesting mechanical, exhaustive ablations, the complexity of this particular design creates a substantial need for more targeted validation. If the authors can clearly explain why each module and its inputs are necessary, with convincing justification where appropriate, I would be able to better understand these design choices.
At present, I remain concerned that the design and its gains may primarily reflect the accumulation of components rather than a clearly established new mechanism. I therefore consider the novelty limited and retain a negative assessment. That said, if the other reviewers and the AC all maintain a positive assessment, I would not object to acceptance.