acceptodds
Under review as a conference paper at ICLR 2027

A Text-Parametrized Metric for Image Similarity

Abstract

Although assessing visual similarity comes naturally to humans, measuring such perceptions remains a challenge. State-of-the-art metrics typically only consider one approach or format for measuring similarity, and do not generalize well to new scenarios. While recent methods begin to compare images in ways beyond overall visual appearance, there are many kind of similarities without an established metric. For example, if users want to compare images with respect to art style, noise pattern, or even the number of cats, would they need to construct a dataset and learn a metric from scratch? Instead, our goal is to learn a single model where the similarity context, in addition to the images, is also a model input. To this end, we learn a text-parametrized perceptual metric that predicts conditional similarity specified in the text prompt. We construct a two alternative forced choice (2AFC) dataset to finetune a perceptual embedding where image representations are influenced by the comparison described in the text. This allows us to compare fine-grained and complex visual attributes with one model. We study our model's capability to measure visual similarity in diverse contexts, demonstrate applications on conditional image retrieval, and compare to existing similarity metrics.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Visual Similarity, Vision-Language Models, Perceptual Similarity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8isYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates text-conditioned image similarity, where a textual description specifies the criterion by which images should be compared. The same pair of images may therefore receive different similarity judgments depending on whether the condition concerns color, shape, layout, or another attribute. The proposed method builds on Qwen3-VL-Embedding-8B. It independently encodes each image under the same textual condition and computes the cosine similarity between the resulting representations. The authors construct comparison samples through generative image editing, collect human judgments of which candidate is more similar to a reference image, and fine-tune the model accordingly. Evaluation covers perceptual similarity judgments, conditional image retrieval, and combinations of multiple conditions.

Critical strengths
  1. A clear research objective with practical relevance Specifying the comparison criterion through text makes the meaning of “similarity” explicit and allows users to express preferences concerning color, structure, style, and other attributes. This formulation can improve the controllability of image retrieval and visual evaluation while making similarity scores easier to interpret.
  2. Data construction tailored to the task, with human supervision The two data construction strategies respectively emphasize fine-grained differences along the same comparison dimension and changes in similarity rankings when the comparison condition changes. These strategies closely reflect the intended model capabilities. Human annotations also provide direct supervision for aligning the model with human perceptual judgments.
  3. Clear empirical improvements under the paper’s evaluation protocol In the main experiment, accuracy increases from 76.80% for the original 8B model to 87.85%, an improvement of 11.05 percentage points (Table 2). This result supports the effectiveness of the proposed training pipeline. Experiments involving multiple image sources and downstream tasks provide additional empirical evidence.
Critical weaknesses
  1. Limited novelty in the core problem formulation and model design Both text-conditioned image similarity and language-controlled image representations have direct precedents, including FocalLens [1] and TPIPS [2]. In particular, TPIPS includes a late-fusion model that uses the same backbone, independently encodes each image under a shared textual condition, and computes cosine similarity. The paper therefore needs to articulate more clearly what substantive advances it offers beyond existing frameworks.
  2. Insufficient direct comparisons with the closest methods The main two-alternative forced-choice (2AFC) experiments lack direct comparisons with closely related methods such as TPIPS [2]. Although CLAY [3] is included in the retrieval experiments, the overall evaluation remains insufficient to establish the proposed method’s advantages over its closest alternatives. Comparisons with the original Qwen model demonstrate the benefit of fine-tuning, but provide limited evidence of the distinctive value of the proposed approach.
  3. Potential limitations in the representativeness of the data and evaluation The main comparison data rely on generative image editing, and samples are filtered for relatively high agreement among annotators. This may favor the chosen editing distribution and cases with clear human consensus, while underrepresenting complex variations in real images, ambiguous cases, and disagreement among observers. Unintended changes to attributes outside the specified condition during editing could also reduce the precision of the supervision.
  4. Limited evidence for generalization and robustness to multiple conditions The out-of-distribution evaluation primarily assesses adaptation to new image sources, which does not sufficiently establish generalization to previously unseen comparison concepts. The experiments involving multiple conditions also exhibit substantial sensitivity to prompt order, suggesting that the model’s interpretation and handling of combined conditions remain unstable.
  5. Insufficient analysis of the sources of improvement The current evidence does not adequately disentangle the contributions of the overall data construction procedure, the two data strategies, the training objective, and the fine-tuning configuration. Consequently, it remains unclear which design choices primarily drive the improvements and which components are necessary. Controlled ablations that match the number of training samples and the training configuration are particularly important for establishing the independent benefits of Strategy 1 and Strategy 2. References [1] FocalLens Hsieh, C.-Y., et al. (2025). FocalLens: Instruction Tuning Enables Zero-Shot Conditional Image Representations. ICLR 2025 Workshop on Foundation Models in the Wild. arXiv:2504.08368. [2] TPIPS Wang, S.-Y., et al. (2026). The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric. arXiv preprint, arXiv:2607.18237. [3] CLAY Lim, S., et al. (2026). CLAY: Conditional Visual Similarity Modulation in Vision-Language Embedding Space. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). arXiv:2604.11539.
Questions for authors

Novelty: What is the substantive contribution beyond related work, particularly FocalLens and the concurrent TPIPS, and what evidence supports this distinction?

Comparisons: Can the authors provide a direct comparison with TPIPS’s late-fusion model under the same 2AFC evaluation protocol to clarify the proposed method’s relative strengths and limitations?

Data reliability: What evidence shows that the reported gains are robust to unintended changes introduced by image editing and to the choice of annotator-consensus threshold?

Generalization and robustness: What evidence supports generalization to unseen comparison concepts and reliable performance under semantically equivalent reorderings of multiple conditions?

Ablation studies: Under matched data and training budgets, which controlled ablations establish the individual contributions of the two data strategies, the training objective, and full-parameter fine-tuning?

Reviewer_KinEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at a core idea: two images can be similar in different ways, depending on what we want to compare. For example, we might be interested in their colours, the pattern on an object, or what a person is doing. The authors propose a model where a text prompt allows to specify what similarity metrics is used to calculate a similarity score between two images. The authors train a model that lets users describe in words what characteristics of the image will be used to assess similarity. They also generate a dataset for 2 alternative forced coice task human annotated The model performs better than the alternatives tested on the authors’ main benchmark. It can also help find images that share particular features in an image retrieval task. However, the results can change depending on how the instruction is written, and much of the evaluation uses AI-generated images. The results are promising, but we should be cautious about assuming the same performance on real-world image collections. Main contribution: A flexible way to compare images according to a user-defined criterion, with promising results primarily demonstrated on the authors’ own benchmark.

Critical strengths

The paper’s main strength is that it proposes an image similarity approach that allows users to specify, in ordinary language, which aspects of the images they want to compare. This makes it generaizable across type of comparison. The comparison between the trained model and the original, untrained model shows a clear improvement, suggesting that the proposed training adds value. The authors test several image types and practical image-search tasks, providing evidence beyond a single experiment. The authors also produced a human annotated dataset.

Critical weaknesses

Format concerns: The citation style is confusing and makes the article difficult to read. Citations appear within sentences without clear separation from the surrounding text, such as parentheses or brackets. This repeatedly interrupts the flow of the argument, as the reader must distinguish citation information from the sentence itself. The placement of tables and figures also does not consistently follow the structure of the text. For example, Table 2 presents results on page 6, before the methods section has been completed. Important tables, particularly those referred to frequently, should appear close to where they are first introduced and discussed, ideally on the same page where the layout permits. A similar sequencing problem affects the explanation of multi-dimensional prompt generation. The authors describe dataset creation and introduce Strategies 1 and 2 on page 4. They then present example images labelled “Multi-dimensional Prompts,” although this concept is not explained until the end of page 6. The concept should be defined before the examples are presented. Figures 6 and 7 likewise appear at least one page before the relevant discussion. Moving them closer to that discussion would improve readability and make the article easier to follow. Figure 7 is also poorly readable. Overall the work needs substantial formatting improvement both of text and of tigures and tables palcement. Litterature concerns: The discussion of related work needs updating and more precise wording. LPIPS and DreamSim remain established baselines, but describing them as “the state-of-the-art image similarity metrics” is too broad without specifying the task and benchmark. The authors should discuss newer relevant methods, particularly DiffSim (Song et al., 2025) and TPIPS (Wang et al., 2026). TPIPS is especially relevant because it conditions image similarity on textual descriptions of comparison aspects, directly bearing on the claimed distinction that the proposed method can adapt to different kinds of comparisons. The statement that LPIPS and DreamSim measure similarity only “across entire images” is also misleading, since LPIPS supports patch comparisons. Their limitation would be more accurately described as the absence of user-specified comparison aspects in their standard formulations. The authors should clarify how their approach differs from recent aspect-conditioned methods and, where applicable, provide an empirical comparison to substantiate the claimed novelty. Methods concerns:

-The multi-dimensional evaluation avoids conflicting attributes. In Section 3.3 (p. 6), the authors group comparison dimensions according to whether they favour image A or image B, then concatenate dimensions favouring the same image. This tests multiple aligned cues, rather than the harder case where one image better matches the reference in colour while another better matches it in shape. The increasing accuracy in Figure 7, across all models, could therefore reflect stronger, redundant evidence rather than an ability to resolve complex comparisons. Even models without text inputs improve as the number of dimensions increases.

-Generalization The generalization of the task is not clear to me since the model is not tested on unseed dimensions of similarities. The paper does not clearly establish that the comparison dimensions themselves are absent from training. Comparing colour in an unseen image domain is different from learning to interpret an entirely new kind of comparison. This distinction matters because adapting to new comparisons is central to the paper’s motivation.

-Filtering disagreement narrows the meaning of the reported accuracy. Section 3.1 retains only unanimous training examples and removes test examples with a 3–2 vote split. This selects comparisons with relatively clear consensus, although the paper motivates similarity as subjective and context dependent. The resulting accuracy describes performance on this filtered subset, rather than on the full range of judgments. The paper should distinguish genuinely defective examples from legitimate differences in perception. It would also be interesting to look at the disagreement data. A better argument and description of this case can be beneficial to the reader, on top of targeted testing.

-The baseline comparison is weak TPIPS is explicitly acknowledged as concurrent work addressing text-conditioned similarity, but no experimental comparison is reported. CLAY appears in the retrieval experiments but not in the main similarity benchmark. Meanwhile, the VLM judges assign separate scalar scores to image pairs, although the target task is a direct choice between two alternatives. Their weak results could partly reflect inconsistent score calibration across separate calls. The author should consider comparing against state of the art models to better understand their advantages in performance.

-Prompt-order sensitivity is a substantial reliability problem. Table 4 reports 39.9 ± 24.0 for “Age, Background” and 34.7 ± 16.1 for “All,” where the variation comes from changing the order of the category names. These are large changes for prompts intended to express the same comparison. For “All,” the reported mean also falls below the pretrained 8B model and CLAY. The authors acknowledge this limitation, but it deserves a better argumentation since language is the main interface through which users control the metric. A more systematic test of attribute order and phrasing woud also be beneficial, if not necessary.

Questions for authors

Generalization to unseen comparison criteria. Are any comparison dimensions entirely absent from training and introduced only at evaluation? Please distinguish generalization to new images or domains from generalization to genuinely new comparison criteria, and clarify which is supported by the experiments. Comparison with relevant baselines. Could you clarify the methodological differences from TPIPS and provide a direct comparison, if feasible given its availability? Why is CLAY evaluated for retrieval but omitted from the main similarity benchmark? Prompt sensitivity. What explains the large variation under attribute-order changes in Table 4? Does this sensitivity persist across semantically equivalent paraphrases, and how does it affect the reliability of text as the interface for specifying comparison criteria? Interpretation of multi-dimensional results. Since Section 3.3 combines dimensions favouring the same candidate, how do you distinguish the benefit of additional aligned evidence from an ability to handle multiple comparison criteria? Have you evaluated conflicting criteria, and how would the intended preference or weighting be defined in such cases? Transfer to real images. Which results establish performance on real photographs with human similarity judgments? Can you report synthetic and real-image results separately and clarify the evidence supporting transfer to real-world image collections? Effect of filtering human disagreement. What proportion of training and test examples is excluded by the agreement thresholds? Have you tested a case in which you do not esclude those

Reviewer_QphnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an image similarity metric and the comparison of attributes from a VLM model. The authors construct approximately 15,000 human-annotated comparisons using reference images from diverse datasets, VLM-generated comparison dimensions, and synthetic image edits. They fine-tune Qwen3-VL-Embedding-8B with a ranking objective and compute similarity using cosine similarity between text-conditioned image embeddings for comparision. Evaluation includes conditional perceptual judgments, prompts combining multiple attributes, and conditional image retrieval.

Critical strengths

Measure the image similarity is a well studied problem, this paper has clear motivation that image pair can be similar in terms of many attributes. The key contribution is the dataset and the dataset construction is thoughtful. The two strategies cover differences within one attribute and comparisons across distinct attributes. The reference sources span photographs, drawings, charts, and restoration-related imagery, providing broader coverage than a narrowly focused benchmark.

Critical weaknesses

The paper reports approximately 15,000 examples but does not clearly provide evaluation-set sizes, unique reference-image counts, per-subset counts, filtering retention rates, or the procedure ensuring that related images and prompt variants stay within one split. Especially, the number of annotators or their agreement rates is missing in the paper. Moreover, the labels are from a forced-choice question seems not right. Two candidates may be equally similar, the requested attribute may be invisible, or different attributes may favor different candidates. Please clarify whether “equally similar” and “cannot judge” responses were available, report the distribution of annotation agreement and filtering rates.

TPIPS is an earlier work on this domain where has a large overlap. The reviewer do se not mean that we have to compare it but the claim of the paper scope should be accurate. The TPIPS has similar but large scale, probably detail discussion and state the difference is necessary.

Questions for authors
  1. How is “human agreement” in Figure 7 computed? Please clarify whether it is directly comparable with model-versus-majority accuracy, provide sample counts and uncertainty estimates, and explain whether the multi-attribute results persist when attributes favor different candidates.

  2. TPIPS addresses the same problem with nearly the same model. It fine-tunes Qwen3-VL-Embedding-8B into a text-conditioned embedding, scores pairs by cosine similarity, and trains on human judgments collected under free-form aspects. Can we have a deep discussion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.