- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies sample-adaptive multi-vector representations for multimodal retrieval and introduces the Sample-Adaptive Multi-Vector Representation(SAMVR)setting and the AdaptiveEmbed framework. Stage I learns grouped, prefix-structured candidate embeddings through Multi-Group Contrastive Learning(MGCL), using SetSim, a symmetric similarity based on optimal one-to-one assignment. Stage II learns an allocation policy from marginal retrieval-AP gains, with MaxSim responses against a source-domain retrieval bank serving as observations. At retrieval time, the policy selects the query’s active vectors, and gallery items activate the corresponding vector positions. Experiments cover eight image–text, video–text, and audio–text benchmarks, supplemented by fixed-capacity comparisons using the same representations, component ablations, cross-domain transfer, alternative retrieval metrics, and oracle analyses. The results show competitive overall retrieval performance with relatively few active vectors on average.
Critical strengths
- A meaningful representation-allocation problem with promising overall empirical results. Treating multi-vector composition and capacity as sample-dependent decisions is a worthwhile research direction. In Table 1, AdaptiveEmbed achieves COCO T2I/I2T mAP of 63.45/61.14, compared with 62.31/60.48 for MetaEmbed(2/4), while reporting fewer active vectors on average. Figure 4(a) and Figure 8 additionally hold the MGCL representations fixed, providing more convincing evidence for the value of learned representation selection.
- Broad cross-modal and cross-domain evaluation with useful experimental controls. The evaluation covers eight datasets and three cross-modal retrieval task families. The image–text model is trained on COCO and transferred to Flickr30K, ADE20K, and OpenImages, retaining the COCO retrieval bank. The controlled MetaEmbed reproduction shares the backbone, training data, and principal optimization settings, and results are averaged over five random seeds. Alternative retrieval metrics and the bank-replacement experiment further strengthen confidence in the overall benefits.
Critical weaknesses
- The benefit of adaptive cardinality is insufficiently separated from adaptive vector composition. The same-representation controls in Figure 4(a) and Figure 8 are useful, but fixed-capacity configurations take equal-length prefixes from both groups, so the two-vector configuration uses two coarse vectors. In contrast, the policy in Appendix A.2 can also select one coarse and one fine vector without changing cardinality. Consequently, the comparison changes both capacity and composition, without explicitly reporting a fixed coarse–fine pair or a fixed-two-vector policy that adaptively selects composition. This leaves unclear how much improvement specifically requires varying the number of vectors per sample, directly affecting the interpretation of SAMVR’s central contribution.
- The independent retrieval benefit of SetSim over MaxSim is insufficiently established. Figure 4(c) demonstrates SetSim’s one-to-one coverage, but this follows from the matching constraint and does not directly establish better retrieval quality. Moreover, the smallest MetaEmbed(2/4)configuration already achieves 100% distinct-candidate-vector coverage, so declining coverage at larger capacities does not adequately explain improvements in the low-capacity regime. Table 3 primarily ablates MGCL and bank feedback, while Table 5 compares candidate capacities and two one-to-one assignment procedures; neither explicitly isolates SetSim’s effect relative to MaxSim. Since SetSim is an important framework component, this gap limits assessment of its methodological contribution and the proposed matching rationale.
Questions for authors
- Clarification—capacity contribution: Do the fixed-capacity comparisons in Figure 4(a) and Figure 8 include a fixed coarse–fine pair and a policy that adaptively selects composition while keeping cardinality fixed at two? Which results distinguish the benefits of changing vector cardinality from those of changing vector composition?
- Clarification—matching contribution: Which controlled comparisons isolate SetSim’s retrieval gains over MaxSim under comparable backbones, group structures, training budgets, and capacity-selection rules? In particular, given that MetaEmbed(2/4)already achieves 100% distinct-candidate-vector coverage, what evidence supports the additional value of the one-to-one constraint for low-capacity retrieval?
2 comments
Follow-Up to Your Review
Dear Reviewer,
Thank you for your constructive feedback. We have carefully addressed your main concerns through additional experiments, analyses, and clarifications in our rebuttal.
We would greatly appreciate it if you could take another look at our responses. If you have any remaining questions or concerns, please feel free to let us know. We would be happy to provide further clarification or additional evidence.
Thank you for your time and consideration.
Best regards,
The Authors
Response to Reviewer Comments.
We thank the reviewer for the careful reading and constructive comments. We appreciate your positive assessment and understand that your main concerns are the respective contributions of adaptive cardinality and vector composition, as well as the role of SetSim in the proposed framework. We address these concerns through additional Oracle analysis to distinguish adaptive composition from capacity allocation, representative retrieval cases, and clarification of SetSim's role in symmetric retrieval. Our point-by-point responses follow.
W1/Q1: Adaptive Cardinality vs. Adaptive Composition.
We agree that adaptive cardinality and adaptive composition are related but distinct. To isolate composition, we conduct an additional two-candidate Oracle experiment with fixed K=2, allowing each sample to select between {g0,g1} and {g0,z1} according to retrieval AP.
With K fixed at 2, adaptive composition improves average mAP over the better fixed composition by 2.18 points on COCO and 1.34 points on Flickr30K, confirming sample-dependent composition preferences.
We further illustrate this with three actual AdaptiveEmbed retrieval cases from the full COCO gallery (123,287 images).
AdaptiveEmbed selects K=2, 4, and 8, respectively, demonstrating sample-specific capacity allocation consistent with our motivation.
W2/Q2: SetSim and Its Contribution.
We clarify that the primary advantage of SetSim is its applicability to symmetric retrieval, rather than necessarily higher retrieval performance over MaxSim. It enables the same multi-vector representation to serve both query and database roles in bidirectional retrieval.
We attribute retrieval improvements primarily to MGCL, which learns more effective structured representations than MetaEmbed, while SetSim provides the corresponding symmetric matching mechanism. SetSim is therefore not claimed as an independent source of performance gains. We will clarify these complementary contributions and narrow our claims accordingly.