acceptodds
Under review as a conference paper at ICLR 2027

AdaptiveEmbed: Sample-Adaptive Multi-Vector Representation for Multimodal Retrieval

Abstract

Multi-vector representations have emerged as an effective paradigm for multimodal retrieval, where each sample is encoded by multiple complementary embeddings to capture fine-grained cross-modal information. However, existing approaches typically use a fixed representation capacity, assigning the same number of vectors to every sample regardless of its content or retrieval difficulty. This fixed-capacity design overlooks the fact that different samples may require different amounts of representation capacity for effective retrieval. In this work, we introduce Sample-Adaptive Multi-Vector Representation (SAMVR), a new problem setting for multimodal retrieval that studies how multi-vector representation capacity can be allocated at the sample level. Under SAMVR, each sample is represented by a content-adaptive embedding set (CAES), whose size is determined according to the sample-specific retrieval utility of additional representation vectors. To instantiate SAMVR, we propose AdaptiveEmbed, a unified framework for learning sample-adaptive multi-vector representations. AdaptiveEmbed learns structured multi-vector representations through Multi-Group Contrastive Learning (MGCL) with the symmetric set-to-set similarity (SetSim), and further employs Utility Policy Optimization (UPO) to determine sample-specific representation capacity via Marginal Utility Allocation (MUA). Experiments across multimodal retrieval benchmarks involving image, text, video, and audio show that sample-adaptive capacity allocation achieves overall better retrieval performance than fixed-capacity multi-vector representations, validating SAMVR as a viable formulation for multimodal retrieval.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-Vector Representation Learning, Multimodal retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GRUCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies sample-adaptive multi-vector representations for multimodal retrieval and introduces the Sample-Adaptive Multi-Vector Representation(SAMVR)setting and the AdaptiveEmbed framework. Stage I learns grouped, prefix-structured candidate embeddings through Multi-Group Contrastive Learning(MGCL), using SetSim, a symmetric similarity based on optimal one-to-one assignment. Stage II learns an allocation policy from marginal retrieval-AP gains, with MaxSim responses against a source-domain retrieval bank serving as observations. At retrieval time, the policy selects the query’s active vectors, and gallery items activate the corresponding vector positions. Experiments cover eight image–text, video–text, and audio–text benchmarks, supplemented by fixed-capacity comparisons using the same representations, component ablations, cross-domain transfer, alternative retrieval metrics, and oracle analyses. The results show competitive overall retrieval performance with relatively few active vectors on average.

Critical strengths
  1. A meaningful representation-allocation problem with promising overall empirical results. Treating multi-vector composition and capacity as sample-dependent decisions is a worthwhile research direction. In Table 1, AdaptiveEmbed achieves COCO T2I/I2T mAP of 63.45/61.14, compared with 62.31/60.48 for MetaEmbed(2/4), while reporting fewer active vectors on average. Figure 4(a) and Figure 8 additionally hold the MGCL representations fixed, providing more convincing evidence for the value of learned representation selection.
  2. Broad cross-modal and cross-domain evaluation with useful experimental controls. The evaluation covers eight datasets and three cross-modal retrieval task families. The image–text model is trained on COCO and transferred to Flickr30K, ADE20K, and OpenImages, retaining the COCO retrieval bank. The controlled MetaEmbed reproduction shares the backbone, training data, and principal optimization settings, and results are averaged over five random seeds. Alternative retrieval metrics and the bank-replacement experiment further strengthen confidence in the overall benefits.
Critical weaknesses
  1. The benefit of adaptive cardinality is insufficiently separated from adaptive vector composition. The same-representation controls in Figure 4(a) and Figure 8 are useful, but fixed-capacity configurations take equal-length prefixes from both groups, so the two-vector configuration uses two coarse vectors. In contrast, the policy in Appendix A.2 can also select one coarse and one fine vector without changing cardinality. Consequently, the comparison changes both capacity and composition, without explicitly reporting a fixed coarse–fine pair or a fixed-two-vector policy that adaptively selects composition. This leaves unclear how much improvement specifically requires varying the number of vectors per sample, directly affecting the interpretation of SAMVR’s central contribution.
  2. The independent retrieval benefit of SetSim over MaxSim is insufficiently established. Figure 4(c) demonstrates SetSim’s one-to-one coverage, but this follows from the matching constraint and does not directly establish better retrieval quality. Moreover, the smallest MetaEmbed(2/4)configuration already achieves 100% distinct-candidate-vector coverage, so declining coverage at larger capacities does not adequately explain improvements in the low-capacity regime. Table 3 primarily ablates MGCL and bank feedback, while Table 5 compares candidate capacities and two one-to-one assignment procedures; neither explicitly isolates SetSim’s effect relative to MaxSim. Since SetSim is an important framework component, this gap limits assessment of its methodological contribution and the proposed matching rationale.
Questions for authors
  1. Clarification—capacity contribution: Do the fixed-capacity comparisons in Figure 4(a) and Figure 8 include a fixed coarse–fine pair and a policy that adaptively selects composition while keeping cardinality fixed at two? Which results distinguish the benefits of changing vector cardinality from those of changing vector composition?
  2. Clarification—matching contribution: Which controlled comparisons isolate SetSim’s retrieval gains over MaxSim under comparable backbones, group structures, training budgets, and capacity-selection rules? In particular, given that MetaEmbed(2/4)already achieves 100% distinct-candidate-vector coverage, what evidence supports the additional value of the one-to-one constraint for low-capacity retrieval?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Comments.

We thank the reviewer for the careful reading and constructive comments. We appreciate your positive assessment and understand that your main concerns are the respective contributions of adaptive cardinality and vector composition, as well as the role of SetSim in the proposed framework. We address these concerns through additional Oracle analysis to distinguish adaptive composition from capacity allocation, representative retrieval cases, and clarification of SetSim's role in symmetric retrieval. Our point-by-point responses follow.

W1/Q1: Adaptive Cardinality vs. Adaptive Composition.

We agree that adaptive cardinality and adaptive composition are related but distinct. To isolate composition, we conduct an additional two-candidate Oracle experiment with fixed K=2, allowing each sample to select between {g0,g1}\{g_0,g_1\} and {g0,z1}\{g_0,z_1\} according to retrieval AP.

DatasetSelect {g0,z1}\{g_0,z_1\}Select {g0,g1}\{g_0,g_1\}
COCO24.04%75.96%
Flickr30K14.05%85.95%
DatasetConfigurationT2II2TAverage
COCOFixed {g0,g1}\{g_0,g_1\}62.9359.3161.12
COCOFixed {g0,z1}\{g_0,z_1\}63.1060.3961.75
COCOTwo-candidate Oracle65.0862.7663.92
Flickr30KFixed {g0,g1}\{g_0,g_1\}78.0574.8976.47
Flickr30KFixed {g0,z1}\{g_0,z_1\}78.1375.2476.69
Flickr30KTwo-candidate Oracle79.2376.8278.03

With K fixed at 2, adaptive composition improves average mAP over the better fixed composition by 2.18 points on COCO and 1.34 points on Flickr30K, confirming sample-dependent composition preferences.

We further illustrate this with three actual AdaptiveEmbed retrieval cases from the full COCO gallery (123,287 images).

COCO IDText Query (Localized Narratives)Rank@K (1–8)Adaptive K
322895These are the sofa chairs ....10→1→5→6→7→11→11→112
453722The image is inside the room ....1174→6→3→1→1→1→1→14
011197In the image we can see there ....844→5→2→2→2→2→2→18

AdaptiveEmbed selects K=2, 4, and 8, respectively, demonstrating sample-specific capacity allocation consistent with our motivation.

W2/Q2: SetSim and Its Contribution.

We clarify that the primary advantage of SetSim is its applicability to symmetric retrieval, rather than necessarily higher retrieval performance over MaxSim. It enables the same multi-vector representation to serve both query and database roles in bidirectional retrieval.

We attribute retrieval improvements primarily to MGCL, which learns more effective structured representations than MetaEmbed, while SetSim provides the corresponding symmetric matching mechanism. SetSim is therefore not claimed as an independent source of performance gains. We will clarify these complementary contributions and narrow our claims accordingly.

AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer,

Thank you for your constructive feedback. We have carefully addressed your main concerns through additional experiments, analyses, and clarifications in our rebuttal.

We would greatly appreciate it if you could take another look at our responses. If you have any remaining questions or concerns, please feel free to let us know. We would be happy to provide further clarification or additional evidence.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_fL6ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies multimodal retrieval with expressive multi-vector representations, focusing on sample-dependent representation capacity. The paper formulates this problem as Sample-Adaptive Multi-Vector Representation (SAMVR), which asks how much capacity should be assigned to each sample and how its representation should be constructed according to the retrieval utility of additional embeddings. To address this problem, the authors propose AdaptiveEmbed, a framework that first learns a structured multi-vector embedding set through Multi-Group Contrastive Learning (MGCL) and then uses a utility-trained policy to select a query-specific subset of vectors. Across eight image–text, video–text, and audio–text benchmarks, the method is evaluated against existing approaches, including MetaEmbed and Qwen3-VL-Embedding, and achieves higher overall average retrieval performance than the evaluated fixed-capacity multi-vector baselines.

Critical strengths
  1. The proposed method provides broad evaluation coverage across different modalities: text-to-video, video-to-text, text-to-audio, and audio-to-text retrieval. This provides a strong indication of the method’s generalization across modalities.

  2. The main results indicate the proposed method is better than the compared alternative single-vector and multi-vector representations for retrieval.

Critical weaknesses
  1. The motivation and core contribution need a clearer intuitive explanation.

The paper proposes choosing representation capacity separately for each query sample. However, the relationship between the benefit of multiple vectors and the benefit of sample-adaptive allocation is difficult to follow. Figure 1 motivates variation in useful capacity, and Figures 4(a) and 8 provide evidence that adaptive allocation improves over the evaluated fixed-capacity settings, but the paper would benefit from concrete examples in the main text showing how additional vectors and sample adaptive capacity help distinguish a correct match from similar candidates, when they improve and when they hurt retrieval. The experiment sections also do not disclose when the proposed method improves or fails on the retrieval examples. Examples or results connecting the learned policy's choices to actual ranking changes would make the importance of sample-adaptive capacity and its contribution beyond existing multi-vector methods more convincing.

  1. The paper also does not provide a complete analysis of computational complexity or end-to-end efficiency.

While multi-vector representations may capture more information and improve retrieval performance, they can also increase matching computation costs and lead to slow retrieval. The efficiency comparison in Figure 5(a) is limited to multi-vector baselines. Single-vector methods such as CLIP, Qwen3-VL-Embedding, and WAVE appear in the retrieval performance tables, but the paper does not provide matched computation latency or memory measurements against them, making the overall comparison incomplete.

  1. The novelty and improvements over MetaEmbed and other multi-vector retrieval methods are insufficiently established.

AdaptiveEmbed builds on grouped multi-vector representation learning and introduces a learned query-dependent allocation policy and SetSim matching. However, its relationship to several relevant predecessors is not discussed, and the following works are absent from the references:

a. Maximal Matching Matters: Preventing Representation Collapse for Robust Cross-Modal Retrieval. ACL 2025.

b. Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval. CVPR 2019.

c. Improving Cross-Modal Retrieval with Set of Diverse Embeddings. CVPR 2023.

These works establish relevant precedents for learned multi-vector representations and set-based matching. The authors should clarify the additional contribution relative to them and explain what is distinctive about the proposed method.

Questions for authors
  1. Could you provide examples showing how the learned policy's allocation decisions affect retrieval rankings? In particular, when do additional vectors help distinguish the correct match, when do they hurt retrieval, and when is it beneficial to stop expanding the representation?

  2. How does AdaptiveEmbed compare with MetaEmbed and single-vector methods such as Qwen3-VL-Embedding and WAVE in end-to-end retrieval latency and memory usage under the same hardware and retrieval protocol? Is the proposed method computationally more expensive? Could you explain how the computational cost scales with the gallery size?

  3. What are the main contributions beyond MetaEmbed, MaxMatch, PVSE, and Improving Cross-Modal Retrieval with Set of Diverse Embeddings? How does SetSim differ from MaxMatch's Hungarian matching in score aggregation and inference, and what evidence shows that the proposed allocation policy improves over existing similar methods? What are the most important contributions of this work stands out compared to many other existing similar works?

  4. Does the proposed method use the same training data as MetaEmbed, and also same network as initilization? Are the comparison fair upon the basis of same or similar data and model except the proposed learning strategies?

9 comments
AuthorsOpenReview ↗

Response to Reviewer Comments (part 1/2)

We thank the reviewer for the careful reading and constructive comments. We understand your main concerns regarding adaptive capacity allocation, retrieval efficiency, methodological novelty, and experimental fairness. We address each concern with additional retrieval examples, end-to-end latency measurements, comparisons with existing methods, and clarification of experimental settings. Our point-by-point responses follow.

W1/Q1: Motivation and Benefits of Sample-Adaptive Capacity.

Our motivation is that the optimal representation capacity varies across samples: additional vectors may improve discrimination, introduce interference, or provide no further benefit. We illustrate this with three actual AdaptiveEmbed retrieval cases from the full COCO gallery (123,287 images).

COCO IDText Query (Localized Narratives)Rank@K (1–8)Adaptive K
322895These are the sofa chairs ....10→1→5→6→7→11→11→112
453722The image is inside the room ....1174→6→3→1→1→1→1→14
011197In the image we can see there ....844→5→2→2→2→2→2→18

AdaptiveEmbed selects K=2, 4, and 8, respectively, demonstrating sample-specific capacity allocation consistent with our motivation.

W2/Q2: Computational Complexity and End-to-End Efficiency.

We measure end-to-end latency on an A800 GPU (batch size 1), including feedback and allocation overhead.

MethodCOCO I2TCOCO T2IClotho T2AClotho A2T
AdaptiveEmbed (~2.3)104.4472.2393.21360.1
MetaEmbed (2/4)85.9352.7877.41298.3
MetaEmbed (16/64)232.92203.25123.11437.7

All values are ms/query.

AdaptiveEmbed reduces latency over MetaEmbed (16/64) by 55.2%/64.5% on COCO and 24.3%/5.4% on Clotho. Feedback and policy overheads are modest: ~20 /~1 ms for image–text and 6.7–10.1/0.9–3.4 ms for audio–text, respectively, and are included above.

AuthorsOpenReview ↗

Response to Reviewer Comments (part 2/2)

W3/Q3: Novelty and Comparison with Existing Multi-Vector Methods.

PVSE, DivE, MaxMatch, and MetaEmbed all learn fixed-capacity multi-vector representations, whereas AdaptiveEmbed addresses sample-adaptive capacity allocation.

MethodMain ChallengeCapacityMatchingAdaptive Allocation
PVSE (CVPR'19)Polysemous semanticsFixedSet-based✗
DivE (CVPR'23)Semantic diversityFixedSmooth-Chamfer✗
MaxMatch (ACL'25)Representation collapseFixedHungarian / Top-k✗
MetaEmbed (ICLR'26)Multi-vector expressivenessFixedMaxSim✗
AdaptiveEmbed (Ours)Sample-dependent capacityAdaptiveSymmetric SetSim✓

We select MetaEmbed as our primary fixed-capacity baseline and additionally compare with the earlier methods on COCO 5K and Flickr30K.

MethodCOCO R@1COCO R@5Flickr R@1Flickr R@5Overall
PVSE38.8068.6551.2578.8059.38
DivE40.5068.9553.9580.1560.89
MaxMatch44.1072.9360.0584.2565.33
MetaEmbed (2/4)51.7972.5469.9086.1570.10
Δ (MetaEmbed − Best)+7.69−0.39+9.85+1.90+4.77
AdaptiveEmbed (Ours)52.6573.2069.7586.0570.41
Δ (Ours − Best)+8.55+0.27+9.70+1.80+5.08

Best denotes the best of the three earlier methods. AdaptiveEmbed achieves a 5.08-point overall improvement.

Although MaxMatch and SetSim both employ one-to-one assignment, their roles differ: MaxMatch uses it primarily for representation learning, whereas SetSim also serves as the actual retrieval similarity, like MaxSim, enabling symmetric matching .

Starting from the motivation of sample-adaptive representation, we design AdaptiveEmbed with MGCL to learn structured candidate representations, SetSim as a symmetric retrieval similarity, and UPO/MUA to allocate sample-specific capacity according to retrieval utility.

W4/Q4: Fairness of Comparison with MetaEmbed.

AdaptiveEmbed and MetaEmbed share the same experimental foundations, as summarized below.

Experimental SettingAdaptiveEmbedMetaEmbedSame
Image–text training dataCOCOCOCO✓
Video–text training dataActivityNet and DiDeMoActivityNet and DiDeMo✓
Text–audio training dataClotho and MACSClotho and MACS✓
Image–text / Video–text backboneQwen-vl-embeddingQwen-vl-embedding✓
Text–audio backboneWAVEWAVE✓
ComponentQ-FormerQ-Former✓

The key differences lie in the learning strategies: MGCL versus single-group contrastive learning, and sample-adaptive versus fixed-capacity allocation. Thus, the comparison isolates the proposed methodological contributions rather than differences in training data or model initialization.

AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer,

Thank you for your constructive feedback. We have carefully addressed your main concerns through additional experiments, analyses, and clarifications in our rebuttal.

We would greatly appreciate it if you could take another look at our responses. If you have any remaining questions or concerns, please feel free to let us know. We would be happy to provide further clarification or additional evidence.

Thank you for your time and consideration.

Best regards,
The Authors

Reviewer_CjzoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to allocate different numbers of representation vectors to individual samples in multimodal retrieval. AdaptiveEmbed first learns grouped candidate embeddings, then uses retrieval feedback to select a subset through a utility-driven policy. Experiments across image–text, video–text, and audio–text retrieval show improved average retrieval performance with fewer active vectors than fixed-capacity alternatives.

Critical strengths
  1. The paper addresses a useful question: how much representation capacity does each sample actually need? The motivation is supported by the observation that adding vectors does not consistently improve retrieval. Treating capacity selection as part of the retrieval process is a worthwhile direction.

  2. The experiments cover eight datasets across three modality pairs. The image–text experiments also evaluate transfer from COCO to three other datasets without target-domain training. These results provide useful evidence of the method’s applicability across different retrieval settings.

  3. The analysis using the same MGCL-trained embeddings while varying the allocation rule is a useful control. It helps separate the benefit of adaptive selection from improvements in representation learning. The ablations on MGCL and retrieval-bank feedback also help explain the contribution of these components.

Critical weaknesses
  1. The novelty of the problem formulation is overstated. Earlier work such as ColBERTer and DocPruner already studies dynamic or content-adaptive vector retention, but these papers are missing from the discussion. The paper needs to establish what utility-driven allocation adds beyond these approaches and include comparisons with adaptive selection baselines.

  2. The improvements over competitive baselines are often modest and are not consistent across retrieval directions. For example, on COCO, the gains over MetaEmbed (2/4) are 1.14 and 0.66 mAP points, while Flickr30K image-to-text retrieval is worse than that baseline. Although results are averaged over five seeds, no variability or significance tests are reported, making these small differences difficult to assess.

  3. The efficiency evaluation measures only part of the retrieval pipeline. Matching time excludes the cost of obtaining bank feedback and running the allocation policy, while every gallery item still stores all eight candidate vectors. End-to-end measurements are needed to establish whether the reduction in active vectors produces a practical speed advantage.

4.Reproducibility is limited by the lack of an accessible implementation; the paper only promises to release code after acceptance. This matters because the evaluation uses reconstructed text descriptions and a nonstandard single-positive protocol. The threshold-calibration procedure is also insufficiently specified, making the reported operating points difficult to reproduce independently.

Questions for authors
  1. What distinguishes the proposed formulation from existing adaptive vector-retention methods, and how does the policy compare with learned pruning under the same average vector budget?

  2. Can you report variability across the five runs and test whether the gains over the strongest comparable baselines are statistically significant?

  3. What is the end-to-end query latency when bank-feedback computation and the selection policy are included?

  4. Can you provide an anonymous implementation, including evaluation preprocessing and threshold-calibration scripts, and clarify which validation queries are used for calibration?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Comments (part 1/3)

We thank the reviewer for the careful reading and constructive comments. We thank your main concerns regarding the novelty of our formulation, performance reliability, retrieval efficiency, and reproducibility. We clarify the distinction from ColBERTer and DocPruner, and address the remaining concerns through additional comparisons, five-run evaluations, end-to-end latency measurements, and an initial code release. Our point-by-point responses follow.

W1/Q1: Novelty and Comparison with Existing Adaptive Methods.

We thank the reviewer for pointing out ColBERTer and DocPruner. Although both employ adaptive vector reduction, they address different retrieval tasks and optimization objectives. ColBERTer targets text retrieval, using whole-word aggregation and learned token reduction to compress document representations. DocPruner targets visual document retrieval, using intra-document patch attention to prune redundant patch embeddings for storage efficiency. In contrast, AdaptiveEmbed addresses sample-adaptive representation capacity allocation for cross-modal retrieval, where capacity is determined according to retrieval utility (see the table below).

MethodTaskMain Challenge
ColBERTerText retrievalRedundant token representations
DocPrunerVisual document retrievalRedundant visual patch representations
AdaptiveEmbedMultimodal retrievalSample-adaptive representation capacity

We further compare their retrieval performance on COCO and Flickr30K:

MethodAvg. TokensCOCO T2ICOCO I2TFlickr T2IFlickr I2T
ColQwen2578.338.9544.9449.2630.21
DocPruner273.436.8140.3245.9027.45
AdaptiveEmbed2.163.4561.1478.2775.72

These methods are primarily designed for document retrieval, where token/patch reduction targets representation redundancy and storage efficiency. However, local visual patches do not necessarily correspond to cross-modal semantic units, and pruning based on intra-document redundancy may discard fine-grained information essential for image–text matching. Consequently, their compression strategies do not directly transfer to cross-modal retrieval, where representation capacity should be determined by retrieval utility.

AuthorsOpenReview ↗

Response to Reviewer Comments (part 2/3)

W2/Q2: Performance Variability and Statistical Significance.

We conduct five independent runs with different random seeds and report standard deviations. The deviations are generally small (0.1–0.4 mAP), indicating stable results across eight datasets and three modality pairs.

Image–Text Retrieval (mAP %)

MethodAvg. TokensCOCO T2ICOCO I2TFlickr T2IFlickr I2TADE T2IADE I2TOpenImages T2IOpenImages I2T
MetaEmbed (2/4)362.3160.4877.5777.2446.6646.4527.7430.03
MetaEmbed (4/8)662.3360.0076.9777.2947.7747.8829.3430.93
MetaEmbed (8/16)1262.1559.3777.1076.4648.2247.7529.4430.46
MetaEmbed (16/64)4062.2759.1277.1076.1048.2247.7529.3330.23
AdaptiveEmbed2.163.45 ± 0.361.14 ± 0.278.27 ± 0.175.72 ± 0.250.21 ± 0.247.27 ± 0.229.90 ± 0.331.09 ± 0.1

Video–Text and Audio–Text Retrieval (mAP %)

MethodAvg. TokensActivityNet T2VActivityNet V2TDiDeMo T2VDiDeMo V2TClotho T2AClotho A2TMACS T2AMACS A2T
MetaEmbed (2/4)360.8652.1347.2644.7532.0330.1714.9310.87
MetaEmbed (4/8)661.4254.3747.8846.1229.6030.9017.3113.30
MetaEmbed (8/16)1261.5155.6047.7146.6230.4231.1418.5514.56
MetaEmbed (16/64)4060.2854.3446.1545.1029.5130.2418.5915.12
AdaptiveEmbed1.961.62 ± 0.456.48 ± 0.348.46 ± 0.246.16 ± 0.332.56 ± 0.131.75 ± 0.220.41 ± 0.318.01 ± 0.2

The additional runs confirm the stability of our results, with strong overall performance using substantially fewer active vectors.

AuthorsOpenReview ↗

Response to Reviewer Comments (part 3/3)

W3/Q3: End-to-End Retrieval Efficiency.

We measure end-to-end latency on an A800 GPU (batch size 1), including feedback and allocation overhead.

MethodCOCO I2TCOCO T2IClotho T2AClotho A2T
AdaptiveEmbed (~2.3)104.4472.2393.21360.1
MetaEmbed (2/4)85.9352.7877.41298.3
MetaEmbed (16/64)232.92203.25123.11437.7

All values are ms/query.

AdaptiveEmbed reduces latency over MetaEmbed (16/64) by 55.2%/64.5% on COCO and 24.3%/5.4% on Clotho. Feedback and policy overheads are modest: ~20 /~1 ms for image–text and 6.7–10.1/0.9–3.4 ms for audio–text, respectively, and are included above.

W4/Q4: Reproducibility and Code Availability.

We provide an initial implementation in our anonymous repository:

https://anonymous.4open.science/r/AdaptiveEmbed--2EDC

We are organizing the complete codebase and will release it at the earliest opportunity.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their thoughtful comments and constructive suggestions. We have carefully addressed the concerns and further strengthened the experimental evidence, methodological analysis, and presentation of AdaptiveEmbed. The major revisions are summarized below.

Summary of Revisions

1. Additional Experiments and Quantitative Evaluations

  • Statistical reliability: Conducted five independent runs with different random seeds and reported standard deviations across eight benchmarks, addressing concerns about performance variability (Reviewer Cjzo).
  • Retrieval case studies: Added three actual COCO text-to-image retrieval cases, illustrating how additional vectors affect positive-match rankings and how the learned policy selects different capacities (Reviewer fL6Z).
  • Adaptive composition: Added a fixed-K=2 comparison between two fixed vector compositions and sample-adaptive selection. Results on COCO and Flickr30K demonstrate that adaptive composition outperforms both fixed configurations (Reviewer GRUC).
  • Related-method comparisons: Added methodological and quantitative comparisons with ColBERTer and DocPruner, as well as PVSE, DivE, and MaxMatch, on relevant retrieval benchmarks (Reviewers Cjzo and fL6Z).

2. Motivation and Problem Positioning

  • Sample-adaptive capacity: Further clarified why additional vectors may improve discrimination, introduce interference, or provide no further benefit, motivating sample-specific capacity allocation (Reviewer fL6Z).
  • Adaptive vs. fixed-capacity methods: Distinguished our formulation from token/patch compression methods and existing fixed-capacity multi-vector representations (Reviewers Cjzo and fL6Z).
  • Cardinality and composition: Used the additional Oracle analysis to clarify their complementary roles and demonstrate the benefit of adaptive composition at fixed cardinality (Reviewer GRUC).

3. Computational Efficiency and Storage

  • End-to-end latency: Added A800 measurements for COCO image–text and Clotho audio–text retrieval, including bank feedback and policy selection (Reviewers Cjzo and fL6Z).
  • Storage efficiency: Clarified that AdaptiveEmbed stores eight candidate vectors per gallery item, compared with 64 for MetaEmbed (16/64), reducing gallery representation storage by 87.5% (Reviewer fL6Z).

4. Methodological Contributions

  • SetSim: Clarified its role as a symmetric retrieval similarity, used both for representation learning and actual retrieval, and distinguished it from existing matching approaches (Reviewers GRUC and fL6Z).
  • MGCL and allocation: Further clarified how MGCL learns structured candidate representations, while UPO/MUA determines sample-specific capacity according to retrieval utility (Reviewers GRUC and fL6Z).

5. Experimental Fairness and Reproducibility

  • Fair comparison: Explicitly clarified that AdaptiveEmbed and MetaEmbed use the same training data, backbone architecture, and pretrained initialization, with differences in their learning strategies (Reviewer fL6Z).
  • Code availability: Provided an initial anonymous implementation and are preparing the complete codebase for public release, addressing reproducibility concerns (Reviewer Cjzo).
AuthorsOpenReview ↗

To the Area Chair

Dear Area Chair,

We sincerely thank the reviewers for their constructive feedback. We would like to clarify an important distinction underlying the concerns raised during the review.

Reviewer Cjzo primarily focused on adaptive multi-vector methods developed for document retrieval, while Reviewer fL6Z emphasized earlier fixed multi-vector representation learning methods. Reviewer GRUC further questioned the respective contributions of adaptive cardinality and vector composition, as well as the role of SetSim. However, the central contribution of our work is sample-adaptive multi-vector representation learning for multimodal retrieval, spanning text, image, video, and audio. Our key objective is to allocate representation capacity according to the retrieval needs of individual samples, rather than simply learning multiple vectors or reducing redundant document representations.

To address these concerns, we have clarified the differences in problem formulation and provided additional quantitative comparisons with the methods highlighted by the reviewers, demonstrating stronger overall retrieval performance. We have also added real retrieval cases illustrating adaptive capacity decisions and ranking changes, a fixed-cardinality composition analysis demonstrating the benefit of adaptive vector selection, and further clarification of SetSim's role in symmetric retrieval. In addition, we report five-run stability results, end-to-end latency measurements, and an initial code release.

We believe our responses clarify the novelty of AdaptiveEmbed and adequately address the reviewers' major concerns. We respectfully hope these clarifications will be considered in the final assessment.

Sincerely,
The Authors

AuthorsOpenReview ↗

Updated Manuscript and Revisions

Dear Area Chair and Reviewers,

Thank you again for your careful reading and constructive feedback.

We have uploaded a revised version of the manuscript addressing the concerns raised during the discussion. The revisions are highlighted in blue for clarity. In particular, we have:

  • clarified the motivation and novelty of sample-adaptive representation capacity;

  • revised Figure 6 to provide intuitive examples of the adaptive decision process;

  • added controlled experiments with PVSE and DivE to isolate the contribution of UPO/MUA;

  • added further analysis on Oracle adaptive allocation and end-to-end retrieval efficiency;

  • added controlled comparisons with existing adaptive multi-vector selection methods, including ColBERTer and DocPruner, using identical candidate representations;

  • clarified the roles of MGCL, SetSim, and UPO/MUA, as well as the fairness and reproducibility of our comparisons.

We hope the revised manuscript and additional experiments address the remaining concerns. We would greatly appreciate it if you could take another look at our revisions, and we are happy to clarify any remaining questions.

Sincerely,

The Authors

AuthorsOpenReview ↗

General Response: Further Clarifications and Additional Experiments

We thank all reviewers for their continued feedback and constructive discussions! In this round, we have further strengthened our manuscript and provided additional evidence to clarify the motivation and methodological novelty of AdaptiveEmbed. The main additions are summarized below.

1. Motivation and Visual Illustration

  • Revised Figure 6 to present three actual COCO retrieval cases, illustrating how vector activation changes retrieval rankings and how AdaptiveEmbed makes sample-specific ADD/STOP decisions.
  • Together with Figure 1, these results explain both why adaptive capacity is necessary and how our method realizes it.

2. Controlled Comparisons with Existing Multi-Vector Methods

  • Added experiments integrating UPO/MUA into the official PVSE and DivE implementations, preserving their original representation architectures and training configurations.
  • Both methods achieve consistent improvements with adaptive allocation over their fixed-capacity counterparts.
  • Added Oracle capacity analysis, showing that different samples and representation architectures exhibit distinct capacity preferences.

3. Comparisons with Existing Adaptive Selection Methods

  • Added controlled comparisons with ColBERTer and DocPruner using identical backbones and candidate representations.
  • AdaptiveEmbed achieves the best results across all four retrieval directions with only 2.1 vectors on average, highlighting the advantage of retrieval-utility-driven allocation over redundancy- or attention-based selection.

Overall Clarification

We believe these additional experiments and revisions address the remaining concerns regarding both multi-vector representation learning and adaptive vector selection. To our knowledge, AdaptiveEmbed is the first work to explicitly formulate sample-adaptive multi-vector capacity allocation for multimodal retrieval across image, text, video, and audio. Our central contribution is to determine which vectors each sample should use and how much representation capacity it needs according to retrieval utility.

We sincerely appreciate the reviewers' feedback and hope these additional results and clarifications establish the novelty and significance of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.