acceptodds
Under review as a conference paper at ICLR 2027

What Lies Within: Diagnostics of Compositional and Sequential Behavior in Long-Context Text Embeddings

Abstract

Embedding evaluation has long relied on indirect external tasks such as retrieval, owing to the opaque, vector-valued nature. We depart from this dependence and present a paradigm that directly analyzes the compositional behavior of the output vector under cumulative synthesis. **RSS** measures the coherence between a text and its embedding representations, while **SRS** measures the order sensitivity across positions in text. **RSS** correlated with downstream retrieval performance (**LEMB**, Pearson , and on NarrativeQA task). **SRS** likewise correlated with **LEMB**, with the negative **SRS** score correlating at . **RSS** required relatively little measurement time compared with **SRS** and **LEMB** ( that of **LEMB**); it can therefore be used to analyze performance by sequence length quickly. Every embedder exhibited a distinguishable **SRS** pattern, and these patterns tracked each model's fine-tuning lineage and base model. Measuring and visualizing order sensitivity-as **SRS** does for single-vector embedders-constitutes, to the best of our knowledge, the first attempt to interpret and structure how the position and order information of a document is preserved at the final representation level. FINESSE, the framework behind these results, will be released as a Python package.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, embeddings, long context, evaluation, probing/analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4xZROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a retrieval-free framework for diagnosing long-context embedding models by composing independent text chunks and computing how the resulting vectors relate to their constituents. It introduces two metrics, RSS, which measures whether a composed embedding stays closer to its constituent chunks than to non-constituents, and SRS, which measures sensitivity to chunk order at different positions in the document. The authors report that both metrics correlate with LongEmbed performance.

Critical strengths
  • The paper moves beyond prior dense retriever analyses, which mostly detect positional bias by measuring per-position contribution to the final vector or downstream retrieval performance when the position of relevant information varies.

  • The authors evaluate their claims over 34 open embedding models of up to 8B parameters, spanning encoder and decoder-based architectures, several pooling strategies, and multiple model families.

  • The model selection process is fully documented, and code and data are promised.

Critical weaknesses

W1: Main concern. Paper is hard to read. It introduces too much terminology before an explanation/intuition of the method, and some concepts of the approach (e.g., cumulative synthesis) are undefined. There are tables without captions in section 3, and the prose is overly verbose and circular. These issues make it difficult to understand the methodology and assess the contribution, and require a substantial revision to address.

W2: The motivation of the method raises a concern that benchmarks such as LongEmbed may be biased, e.g., when relevant information is concentrated near the beginning of documents. This is never tested. Also, RSS and SRS are validated by correlation with LEMB. If LEMB has the suspected bias, agreement does not entail that proposed metrics capture holistic long-context understanding.

W3: The main metric, RSS, is defined as 500x(TD + BU). The “high correlation” between TD and BU and RSS is presented as evidence that the metric is “methodologically justified”. This correlation holds by construction and does not support that both components are needed. In fact, Table 11 suggests that the two are redundant (Pearson correlation avg 0.86 across L, 0.93 at L = 9).

Questions for authors

Table 4 reports avg wall-clock time per model of 366min for SRS, and 14min for LEMB. Could the authors clarify when one would prefer this metric over evaluating on LEMB directly? Or how does the metric compliment the LEMB evaluation?

Reviewer_wp9iOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies long-context text embeddings. The paper constructs inputs by concatenating text chunks and varying their order and position. They introduce two diagnostic metrics, Relative Separation Score (RSS), and Sequence Reversal Score (SRS). They evaluate these metrics across 34 embedding models and examine their relationship with downstream retrieval performance.

Critical strengths

The evaluation covers a broad set of embedding models.

Some analyses are interesting and provide useful observations.

Critical weaknesses

My main concern is whether the experimental construction supports the conclusions about the theme. Concatenating different chunks provides a controlled diagnostic, but its relationship to naturally long documents is unclear. Such documents often contain dependencies across sections that this construction does not capture. The main experiments also use a context limit of 8k tokens, which is not quite long. I am therefore not convinced that the findings generalize to the long-document settings motivating the paper.

I think the interpretation of RSS is insufficient. The similarity statistics quantify a geometric separation, but it is unclear how directly that separation measures content preservation. Scores could improve through changes in embedding geometry without a corresponding improvement in the information retained. The reported correlations with retrieval performance are useful, but do not resolve this ambiguity. A simple baseline, such as a CBOW-based document representation might do this as well.

I think SRS could measure a response to sequence reversal, but the connection between that response and semantic understanding remains unclear. A model could be sensitive to the positions of chunks without representing the semantic relationships between them. This matters particularly when the paper interprets SRS patterns as evidence of holistic understanding.

Questions for authors

See weaknesses.

Reviewer_ox5fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present robust separation score (RSS) and sequence recognition score (SRS), combined called FINESSE, a tool that evaluates the compositional behavior of text embedding models. Using the LEMB evaluation suite as validation for these scores, they show that SRS and RSS are more efficient than LEMB across a large amount of models.

Critical strengths

The proposed RSS metric's real value is in speed relative to the existing LEMB suite.

I found the presentation of Section 3 easy-to-follow and well-motivated.

The time cost analysis is interesting and promising.

Critical weaknesses

The primary issue is that the paper frames the need for new metrics as a way to replace external evaluation of text embeddings, then evaluates the new metrics by correlating them with external evaluations of text embeddings (LEMB). The circularity of that argument means there is no real validation for RSS and SRS.

Experiments are somewhat limited, evaluating only one dataset at a fixed arbitrary chunk size of 500 tokens.

Minor:

  • RSS and SRS should be defined in abstract, and again in the introduction.

  • I didn't really follow a lot of the motivation of the introduction. I feel it could simply start with an expanded discussion of the paragraph starting at 050 and should probably be framed more specifically around long-context embedding (as title suggests).

  • 082: This is an incomplete sentence.

Questions for authors

How might RSS and SRS be evaluated independently from LEMB scores? Or what is the benefit of using RSS/SRS over LEMB (other than relative speed)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.