acceptodds
Desk rejected at ICLR 2027

CIM-Embed: Learning Instruction-Conditioned Multimodal Embeddings for Clustering

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal objects can require different partitions under different instructions, as when products in an e-commerce catalog are grouped by color, material, or both. Retrieval-oriented embeddings may not recover these partitions, while generative clustering often relies on intermediate descriptions or iterative decisions. This paper presents CIM-Embed, a framework for learning instruction-conditioned multimodal embeddings for clustering. Supervised contrastive learning uses same-group positives and negatives from other groups, optionally including other tasks; heterogeneous structured batches accommodate tasks with different group counts. At inference, the model maps each object's image, text, and instruction directly to an embedding; HDBSCAN then forms a partition without the ground-truth cluster count. We construct ProductMIC from product metadata in Amazon Reviews 2023, comprising training data and an evaluation benchmark for instruction-conditioned clustering. The benchmark contains 5,373 products and 449 single-attribute and compositional grouping tasks spanning 50 leaf categories excluded from our training. With full-dimensional embeddings and HDBSCAN, direct within-task SupCon training achieves a mean per-task V-measure of 95.01, compared with 53.06 for WeMM 9B, the strongest general-purpose embedding baseline evaluated. Frozen product-trained encoders also improve out-of-domain conditional grouping, while controlled transfer experiments distinguish partition transfer from instruction response. We will release the code and dataset upon acceptance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Clustering, Instruction-Conditioned Embeddings, Controllable Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hiBmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Products on eCommerce platforms are often tagged with multiple attributes. Several applications may require the products to be clustered (or grouped) on the basis of a given attribute. The paper poses it as instruction-conditioned multi modal clustering problem. It proposes CIM-Embed, which fine-tunes Qwen3-VL-Embedding with supervised contrastive loss. The object/product embeddings can then be clustered using existing clustering algorithm HDBSCAN, without specifying the cluster count. The proposed method presents significant benefits over general purpose embeddings (e.g., WeMM-9B) as well as other domain specific baselines including GCPM, ABC, CLAY, ITGC and Multi-Stub. The results are also reported on Cards and CLEVR-4 benchmark datasets to demonstrate transferability to other domains.

The paper also presents the ProductMIC dataset from Amazon Reviews 2023, with VLM-extracted and metadata-verified attribute labels. The proposed dataset has 449 tasks over 5,373 products in 50 held-out leaf categories.


Updated the score based on the discussion during rebuttal.

Critical strengths
  1. The paper addresses a real-world problem concerning multiple eCommerce platforms.
  2. The ProductMIC dataset can be useful for researchers or practitioners working in this domain.
  3. The proposed method achieves significant gains over general-purpose embeddings including WeMM-9B, Qwen3-VL8B and Gemini Embedding 2. For instance, WithinSupCon variant achieves V-measure of 95.01, as compared to 53.06 for WeMM-9B.
  4. The proposed method achieves significant gains over baselines including GCPM, ABC, CLAY, ITGC, Multi-Stub. For instance, Direct Within reaches V=95.01 and ARI=0.9238 as compared to 74.85 and 0.5786 for GCPMG3.
  5. The results are also presented where the frozen product-trained encoders are applied for conditional grouping on other tasks including Cards and CLEVR-4. For instance, Direct Within achieves V=88.55 (baseline is 56.36) on Cards and V=66.46 (baseline is 42.07.36) on CLEVR-4 with Qwen-8B base model.
Critical weaknesses
  1. Useful baselines are missing: (i) Fine-tuning the same backbone with simpler objectives, such as attribute classification heads, triplet/conditional-similarity losses, or an instruction-concatenated InfoNCE variant that is not confounded with sampling. (ii) Directly predicting attribute values with the VLM and grouping by exact match. This corresponds to the annotation pipeline used for label annotation, and it may be a strong baseline. (iii) Finetuning ABC and CLAY on ProductMIC-Train.
  2. CIM-Embed receives image, title and description, while ABC, CLAY, ITGC and Multi-Sub receive image + instruction only (#1619-24). Titles and descriptions often state color, size and so on directly, so this alone could explain much of the gap. A text-only and an image-only CIM-Embed ablation would be helpful.
  3. Possible dataset biases: (i) 7.5% of evaluation labels are model-supplemented. No human audit or error-rate estimate is reported. (ii) The same model family is used to annotate a subset of dataset.
  4. Missing train and inference times.

General suggestions:

  1. The paper overall is well written; however, it is appendix heavy. Please consider re-structuring so that the key points are in the main paper itself.
  2. Consider adding real world benefits for a wider audience.
Questions for authors
  1. Are the numbers available for GCPM, ITGC and Multi-Sub on Cards, CLEVR-4 and CIFAR10-MC under the same clustering protocol as the CIM-Embed transfer runs? This may provide a good baseline for transfer leaning.
  2. How does the model do on held-out attribute names, paraphrased instructions, and held-out attribute combinations?
  3. Please comment on decoupling the loss from sampling, for example InfoNCE with partition-structured batches, or SupCon with random batches?
  4. Consider adding more baselines -- please refer to 1 above (especially 1(ii)). Not all of them are required, but some may help strengthen the paper.
  5. How do results differ on Details-verified labels only? An ensemble annotator should be considered if the results vary significantly.
Reviewer_nheiOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a framework that adapts multimodal embeddings for instruction-conditioned clustering using supervised contrastive learning. It constructs positive and negative pairs according to each instruction and compares contrastive learning within and across tasks. The paper also introduces a benchmark and evaluates the learned embeddings on product grouping and transfer to other domains.

Critical strengths
  • The paper presents a method for clustering multimodal objects according to instructions, allowing the same collection to be organized by different criteria. The accompanying benchmark provides a useful setting for evaluating this capability.

  • The experiments examine both clustering performance and responsiveness to grouping instructions, providing a more informative assessment of the proposed method.

Critical weaknesses
  • The advantage over simpler alternatives is not sufficiently established. Attribute extraction followed by grouping is a direct baseline, but its accuracy and computational cost are not compared with those of the proposed approach.

  • The flexibility beyond the evaluated grouping criteria remains unclear. Experiments cover at most three attributes and do not establish whether new grouping attributes can be handled without additional supervision or retraining. Combining more attributes may also leave groups with too few examples to satisfy the sampling requirement. These limitations weaken the evidence that the approach can support a broader range of grouping instructions.

Questions for authors
  • A straightforward alternative is to extract the attributes specified by the instruction from each object and then group objects with matching attribute values. Comparing accuracy and computational cost against this baseline would help establish the advantage of the proposed embedding-based clustering approach.

  • The contribution beyond standard task-specific fine-tuning remains unclear. InfoNCE fine-tuning already achieves strong performance, while the SupCon comparisons change both the loss function and sampling strategy, as acknowledged in Appendix I.3. Separate ablations on the loss function and sampling strategy would help clarify their individual contributions.

  • Scalability to more complex attribute combinations remains unclear. Experiments cover at most three attributes, and the sampling strategy requires multiple objects per group. As more attributes are combined, fewer objects may share the same attribute combination, leaving some groups with too few examples for training. The paper should evaluate how increasing the number of attributes affects performance and how many groups are excluded by the sampling requirement.

  • It remains unclear whether the model can handle new grouping attributes without additional data and retraining. If each new attribute requires explicit supervision, this would limit the practical flexibility of the proposed method.

Reviewer_UtVQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces instruction-conditioned clustering: given a set of objects and an instruction specifying a grouping attribute, the model must recover the corresponding partition without knowing the number of clusters. CIM-Embed fine-tunes Qwen3-VL-Embedding with supervised contrastive learning, using the instruction to define positive pairs and structured batches to accommodate different groupings. The paper also introduces ProductMIC, a benchmark of 449 tasks over 5,373 Amazon products. CIM-Embed achieves the best task-level V-measure of 95.01, compared with both general-purpose baselines and alternative similarity objectives InfoNCE.

Critical strengths

The task is a contribution: recovering a full partition from a natural-language instruction, without knowing cluster count, goes beyond retrieval or conditional similarity. The paper distinguishes these problems carefully.

ProductMIC is carefully constructed: its 50 evaluation leaf categories are excluded from training, with no overlap at leaf or product level.

The results hold across several checks: the method beats InfoNCE on three backbones, with fixed clustering parameters (90.22), and leaf-disjoint cross-validation (95.15).

Critical weaknesses

The paper does not show why this method is needed. Since the ground-truth groups are determined by attribute labels, the task could be solved by predicting attributes and grouping products accordingly. But the paper does not test this simpler approach, either with a classifier or a predict-then-group pipeline. Without this comparison, the results show that the method works, but not that it is better than simpler alternatives.

The main comparison is also unfairly confounded. Although the test categories are held out, the fine-grained attribute information are already seen during training. CIM-Embed benefits from this information, while the baselines do not get the same advantage. The large gap in V-measure (95 vs. 53) therefore cannot be attributed to the method alone.

There is also a practical limitation that the evaluation does not address. Because the embedding depends on both the product and the instruction, every new instruction requires re-embedding the entire catalog. This makes it difficult to build an index once and reuse it for different queries. A simpler approach could embed each product once, predict its attributes, and group or filter products at query time. Since the benchmark uses small product pools (10–98 products), this cost is not visible.

The paper should explain when its approach is worth this extra cost and where it offers a clear advantage over simpler alternatives, in real world applications.

Questions for authors

Beyond the concerns raised in the Weaknesses section, I’m also curious how triplet loss or Multi-Similarity loss would perform under the same partition-structured batches. Comparing them with the proposed objectives could help determine whether the benefits of this recipe extend beyond softmax-based losses.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Additional cross-domain comparisons under the original evaluation protocol

We have completed the ITGC and Multi-Sub comparisons on Cards, CLEVR-4, and CIFAR10-MC. Each method and dataset independently selects one parameter setting from the original CIM search space: 12 candidates on Cards and 48 each on CLEVR-4 and CIFAR10-MC. Selection maximizes V-measure averaged equally over instructions, using the target labels as in the original protocol. Ties are resolved by mean coverage and then grid order. All samples are scored, with the remaining noise label treated as a shared predicted cluster.

MethodCardsCLEVR-4CIFAR10-MC
CIM-Embed88.5564.1931.76
ITGC43.5643.128.75
Multi-Sub24.2116.361.74

CIM achieves higher average V-measure than both methods on all three datasets under this protocol. These results replace the earlier pending entries for Multi-Sub; they are distinct from the fixed-ProductMIC-parameter transfer diagnostic.

AuthorsOpenReview ↗

Clarifying the contribution and deployment workload

Thank you for considering the additional evidence. We would like to clarify two distinctions relevant to the remaining concerns.

First, the contribution is learning representations for instruction-defined partitions, together with a benchmark that makes this behavior measurable. Training is central to this contribution: the same product pair can be positive under one instruction and negative under another. The controlled ablations measure the additional effect of particular loss and sampling choices within this formulation. They do not remove instruction-conditioned grouping supervision, so their small differences do not establish that the formulation itself adds little. The original comparison with off-the-shelf embeddings measures complete-system performance; the supervised attribute-generator comparison addresses a different question, with both systems receiving ProductMIC supervision.

Second, a new instruction requires re-encoding the products to which it applies, not necessarily the entire catalog. In our intended category workflow, operators maintain relatively stable grouping criteria. With the encoder, instruction, and product inputs unchanged, existing embeddings are reusable; new or edited listings require encoding. Supporting different instructions across categories does not imply repeatedly re-encoding the whole catalog for every query.

This workflow also explains the choice of representation. The required outputs are groups and instruction-conditioned similarity rankings. CIM provides both without requiring explicit, standardized attribute values as an intermediate output. An attribute-based system instead needs to reconcile predicted values before grouping; using those values in shopper or seller interfaces introduces additional naming conventions. We present this as a workflow motivation, not a measured estimate of maintenance savings. On the measured first-request workload, with both models loaded on the same A100, CIM returned a partition in a median 3.82 seconds per pool versus 7.78 seconds for the supervised generator.

We therefore ask that the contribution be assessed as an instruction-conditioned grouping framework and benchmark, with controlled comparisons of alternative implementations, and that its deployment cost be evaluated against the category workflow it is intended to support.

Reviewer_hiBmOpenReview ↗

Followup

Thanks for the responses. I have updated the score accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.