- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a retrieval-free framework for diagnosing long-context embedding models by composing independent text chunks and computing how the resulting vectors relate to their constituents. It introduces two metrics, RSS, which measures whether a composed embedding stays closer to its constituent chunks than to non-constituents, and SRS, which measures sensitivity to chunk order at different positions in the document. The authors report that both metrics correlate with LongEmbed performance.
Critical strengths
-
The paper moves beyond prior dense retriever analyses, which mostly detect positional bias by measuring per-position contribution to the final vector or downstream retrieval performance when the position of relevant information varies.
-
The authors evaluate their claims over 34 open embedding models of up to 8B parameters, spanning encoder and decoder-based architectures, several pooling strategies, and multiple model families.
-
The model selection process is fully documented, and code and data are promised.
Critical weaknesses
W1: Main concern. Paper is hard to read. It introduces too much terminology before an explanation/intuition of the method, and some concepts of the approach (e.g., cumulative synthesis) are undefined. There are tables without captions in section 3, and the prose is overly verbose and circular. These issues make it difficult to understand the methodology and assess the contribution, and require a substantial revision to address.
W2: The motivation of the method raises a concern that benchmarks such as LongEmbed may be biased, e.g., when relevant information is concentrated near the beginning of documents. This is never tested. Also, RSS and SRS are validated by correlation with LEMB. If LEMB has the suspected bias, agreement does not entail that proposed metrics capture holistic long-context understanding.
W3: The main metric, RSS, is defined as 500x(TD + BU). The “high correlation” between TD and BU and RSS is presented as evidence that the metric is “methodologically justified”. This correlation holds by construction and does not support that both components are needed. In fact, Table 11 suggests that the two are redundant (Pearson correlation avg 0.86 across L, 0.93 at L = 9).
Questions for authors
Table 4 reports avg wall-clock time per model of 366min for SRS, and 14min for LEMB. Could the authors clarify when one would prefer this metric over evaluating on LEMB directly? Or how does the metric compliment the LEMB evaluation?