acceptodds
Under review as a conference paper at ICLR 2027

Semantic Fragment Similarity Representation Learning for Information Retrieval

Abstract

We introduce Semantic Fragment Similarity (SFS), a similarity metric designed to enhance representation quality by partitioning embeddings into non-overlapping fragments, computing fragment level similarity, and aggregating these local scores. Conventional similarity metrics compute relevance using the global vector as a single unit. This process flattens and entangles multi-faceted semantic features and dilutes the fine-grained alignment signals crucial for accuracy. By inducing fragments to specialize in distinct semantic roles, SFS drives the substantial gains in retrieval performance across a wide range of models, tasks, and architectures when applied in both training and inference. Further, we find that a single embedding fragment trained with SFS, comprising just 2% of the total dimensions, outperforms the entire global embedding on specific classification tasks. Ultimately, SFS can be directly integrated as a replacement for conventional similarity metrics, without architectural modifications or complex computational overhead and it opens up new avenues for building more structured and interpretable embedding models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Information Retrieval, Representation Learning, Sentence Embeddings, Fragment Similarity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eqKXOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Semantic Fragment Similarity, which splits an embedding into disjoint, contiguous fragments, computes cosine similarity for each fragment, and averages the scores. It is used for both contrastive training and inference. The authors report nDCG@10 gains of 1 to 3% on NanoBEIR across five encoders and two small decoders, and claim that fragments specialize in distinct semantic roles.

Critical strengths

The method is simple and requires no architectural change. As Appendix B shows, it reduces to a dot product after normalizing each fragment, so it remains compatible with standard nearest-neighbor indexes. The evaluation covers several encoder and decoder backbones. These points do not compensate for the weaknesses below.

Critical weaknesses
  1. The method has no grounding and is a fixed instance of a learnable family. Appendix B shows that SFS is the cosine similarity between embeddings normalized per block of contiguous dimensions. The paper never explains why contiguous index blocks should align with semantic aspects. Encoder dimensions have no semantic order, and any permutation would yield different fragments. The paper chooses the number of fragments via search, and the optimum changes across models. Table 11 shows fluctuations in noise order rather than a clear trend. More importantly, uniform averaging over fixed blocks is a hand-crafted choice. Learned alternatives that keep the dot product form, and therefore remain indexable, are natural baselines: learned weights per fragment, or a learned linear projection followed by normalization. The authors acknowledge in the limitations that uniform averaging is an assumption, but no learned variant is compared. Related work on multi-vector representations, product quantization, and normalization or whitening against anisotropy is also missing. This matters because without grounding or a comparison with learned similarity, the contribution reduces to a fixed normalization layer.

  2. The gains are marginal and not statistically supported. Improvements range from 0.002 to 0.014 in nDCG@10. Each configuration is trained once, with no seeds or significance tests, on NanoBEIR subsets of about 50 queries each. The text states that all fragmented configurations outperform the baseline, but Table 1 contradicts this for BERT and ModernBERT at (12,64), and Table 2 shows the same for average pooling. The only baseline is the same model trained with global cosine. In Table 4, the Euclidean and dot product baselines are trained without normalization, so the large gains there come from normalization, not from fragmentation. The evidence for semantic specialization is also weak. It relies on single-fragment nearest-neighbor classification with scores near 0.5 on binary tasks, which is consistent with the known redundancy of embedding dimensions. The kurtosis above 300 in Figure 2 points to extreme outlier dimensions rather than disentanglement. Finally, Table 8 reports almost twice the latency on SciFact, contradicting the claimed equivalence to a standard dot product. This matters because the empirical results are the only support for the method.

Questions for authors

How does SFS compare with learned similarity variants that remain indexable, such as learned weights per fragment or a learned projection followed by normalization?

Are the gains significant across several training seeds, and do they persist when dimensions are randomly permuted before fragmentation?

Reviewer_8U3wOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Semantic Fragment Similarity (SFS), which partitions embeddings into non-overlapping fragments, computes cosine similarity between corresponding fragments, and averages their scores. SFS replaces global cosine similarity during contrastive training and retrieval. Experiments across several encoder and decoder models report improved retrieval performance under suitable fragmentation configurations. Additional analyses investigate coordinate distributions and fragment-level classification performance, which the authors interpret as evidence of semantic specialization.

Critical strengths
  1. The method is simple and compatible with existing retrieval pipelines. SFS requires no additional model parameters, and its formulation permits retrieval through standard inner-product search after fragment-wise normalization.

  2. The experiments cover several relevant settings and are clearly presented. Comparisons across models, pooling strategies, and downstream tasks provide useful evidence about the method’s applicability. The visualizations make the observed representation differences easy to inspect.

Critical weaknesses
  1. The entropy analysis does not sufficiently support the proposed semantic interpretation. Figure 2 measures entropy after applying softmax to normalized embedding coordinates. Negating all embeddings preserves every global and fragment-wise cosine similarity, and therefore the retrieval results, but can change this entropy. Lower entropy alone does not establish improved semantic organization or reduced information dilution.

  2. The distinction between coordinate concentration and semantic specialization remains insufficiently established. The reported coordinate statistics and strong performance of selected fragments are suggestive, but do not by themselves demonstrate that different fragments encode distinct, complementary semantic roles. This limits the strength of the paper’s explanation for its empirical gains.

Questions for authors
  1. How should the coordinate-based entropy in Figure 2 be interpreted, given that a simultaneous sign reversal of all embeddings preserves cosine similarities and retrieval rankings but can change the entropy substantially? What evidence connects this statistic to semantic organization rather than the chosen coordinate representation?

  2. Could you clarify how the analyses distinguish semantic specialization from coordinate concentration or removal of distracting dimensions?

Reviewer_e2BrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Semantic Fragment Similarity, a simple metric that replaces global cosine similarity with an average of fragment-level similarities. The goal is to prevent the flattening of semantic features and encourage fragments to specialize. The method improves performance across several models and embedding tasks at suitable fragment sizes. SFS can also be implemented with a single standard ANN index, which makes it practical to integrate into existing retrieval systems. My main concerns are about the reliability of the smaller improvements and the interpretation of the analysis.

Critical strengths
  • The proposed method is very simple, requiring no changes to the model architecture and adding no new parameters. This makes it easy to adopt.
  • Appendix B shows that SFS is compatible with standard ANN search. Independently normalizing each fragment and concatenating them reduces SFS to a standard inner product up to a constant factor. Multiple ANN indexes are not needed.
  • The experiments cover multiple encoder and decoder models, pooling strategies, and downstream tasks. The finding that a small fragment can outperform the entire embedding on a classification task is also interesting and might spark future research.
Critical weaknesses
  • Some improvements are small, and their statistical reliability is unclear. For example, BERT improves little in Table 1. The paper fixes random seeds but does not report variation across training runs or significance tests.
  • The interpretability claim is only partially supported. Section 6.2 suggests that fragments differ in their usefulness for toxicity and counterfactual classification, but it does not establish distinct semantic roles across fragments. The entropy and kurtosis analysis measures coordinate concentration, which is different from semantic disentanglement. A systematic analysis of what each fragment learns is missing.
  • The ANN equivalence is convincing, but the scalability experiment is limited to three Nano datasets. Appendix D also clarifies that the main retrieval evaluation uses NanoBEIR. Results on full retrieval corpora, with ANN recall and latency measured under matched settings, would better support the large-scale retrieval claim.
  • The interpretation of the dot-product and Manhattan results in Table 4 needs clarification. With ordinary dot product or negative L1 distance, averaging fragment scores is just the global score divided by the number of fragments. This changes the effective contrastive loss temperature. Without details on normalization and a matched-temperature comparison, these gains may come from score scaling.
Questions for authors
  • Can the authors report results across multiple training seeds and on full BEIR corpora? How is fragment size selected without using the test results?
  • For interpretability, wonder if the authors can label fragment roles across several semantic attributes, and compare against the same fragments from a model trained with global similarity?
  • Do the dot-product and Manhattan gains remain after matching the global baseline's score scale?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.