acceptodds
Under review as a conference paper at ICLR 2027

Better Contrast Decisions, Worse Ranking? How Validation Selects Readouts for Text Embeddings

Abstract

A scoring function can rank a passage above its negated counterpart yet place it below other documents in a collection. We study how to select functions that improve this contrast while retaining the benchmark answer's ranking performance. Using frozen text embeddings, we hold each candidate bank fixed within a selection experiment and vary the documents and criteria used for validation. On NevIR, inspection after selection finds candidates that improve contrast accuracy with no decrease in measured test ranking. Yet contrast-only validation repeatedly selects functions with substantial ranking losses. Including retrieved competitors and prioritizing ranking quality reduces these losses while retaining contrast gains. Competitor identity matters even at a fixed list length, and more labeled contrasts do not replace missing background comparisons. The same selected set can also improve average fixed-list reranking while lowering average full-corpus ranking. On NFCorpus, the trained candidates offer only small gains in distinguishing relevance grades, leaving little improvement for a selector to recover. These results distinguish gains available among the trained candidates from gains recovered by validation, and show why a selected scoring function must be evaluated in its intended retrieval setting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
dense retrieval, text embeddings, model selection, negation-aware retrieval, reranking, metric learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_W61SOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how validation design affects which scoring function gets selected for frozen text embeddings. The authors train low-rank residual adapters (“readouts”) on top of fixed embeddings from six encoders. Each readout is trained with a hinge loss so the correct passage outscores its negated twin on NevIR, plus a term that keeps background-document scores close to the original cosine scores. Varying the hyperparameters gives a fixed “bank” of 16 readouts plus raw cosine (30 banks over encoders and seeds). The paper then asks which validation setup picks a readout that improves contrast accuracy without hurting corpus ranking.

The main findings:

  1. Every bank contains readouts that improve contrast without ranking loss, yet contrast-only validation consistently picks readouts with large MRR losses (−0.18 MRR@10 on average).
  2. Selection that uses ranking, especially with retrieved rather than uniformly sampled background documents, avoids these losses while keeping part of the contrast gain. At a fixed list length, which background documents validation sees matters, not just how many.
  3. On NFCorpus, the trained candidates offer only marginal gains in ranking document, so there is little for validation to recover.
Critical strengths

Freezing the bank cleanly separates what training made available from what validation recovers. That design is the main methodological idea.

Critical weaknesses

Thin novelty, the central result is mostly expected. “Validate on the metric you care about” is standard practice. The training loss never compares the answer with background documents (§3.2 admits this), so contrast-only selection losing ranking is expected. Retrieved hard negatives beating uniform ones is also well known.

Questions for authors

Does the selection problem survive a ranking-aware training loss? The training objective used in the paper compares the answer only with its negated twin.

Reviewer_GoSeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies the effects of various validation approaches in selecting scoring functions for frozen embeddings. They find that including a ranking measure reduces loss on the Negation in Neural Information Retrieval benchmark.

Critical strengths

None

Critical weaknesses

The introduction is poorly-written. NevIR is introduced in a clunky manner, and there is no formal definition of the problem the authors are trying to study or their motivation for doing so. The idea of a readout is vaguely hinted at for seemingly no reason. Figure 1 is thrown in with no introduction or interpretation. Only three papers are cited despite the vast amounts of relevant contrastive learning literature.

This presentation throughout is very poor. 3.2 offers no motivation whatsoever. 3.4 offers no motivation whatsoever. There is no clear contribution -- line 051 reads "the fixed-bank design separates what training makes available from what validation recovers" with no explanation of what the fixed-bank design is in their work, why it differs from other relevant work, what the authors expect to learn, or how it might impact our understanding of text embeddings.

Every single sentence is written as a punchy, turn-heavy takeaway, to the point that the entire work is unreadable.

Examples

  • 045: "Training creates a candidate bank; validation chooses one member; test evaluation measures its outcome."
  • 080: "Test coordinates are inspected after validation selection, never used to make it."
  • 241: "These post-selection results establish that useful alternatives were observed within each bank, not merely somewhere in the pooled point cloud of Figure 1."
  • 427: "The contribution is to distinguish a missed available gain from limited gains in the candidate bank, rather than to identify the best function the embeddings could support."
Questions for authors

None

Reviewer_hhHWOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper investigates a practical problem in text retrieval systems. A new scoring function can learn to favour the correct passage over an almost identical but negated version while also worsening the answer’s ranking in the full corpus. Improving a local comparison between two texts therefore does not guarantee better performance in a real retrieval setting, where the answer competes with many other documents. The authors compare different validation views and three selection rules: maximising contrast accuracy, maximising ranking performance, or maximising contrast accuracy subject to matching or exceeding the original cosine baseline’s validation ranking performance. Since the candidate bank remains fixed, differences can be attributed to selection rather than to the training of different candidates. The results show that better discrimination between a statement and its negation does not guarantee better full-corpus ranking. Including competing documents in validation and prioritising ranking helps reduce losses while retaining some of the gains in contrast accuracy.

Critical strengths

I appreciate the clean methodological choice of freezing the same candidate bank before comparing selection rules. This makes it possible to isolate the effect of validation-based selection.

Critical weaknesses
  • I wonder how strongly the conclusions depend on the small low-rank readouts and the specific regularisation grids.
  • As the authors acknowledge, MRR tracks the position of the designated answer. A document ranked above it may still be relevant, so some of the measured deterioration may not correspond to an actual retrieval error from the user’s perspective.
  • The experimental design is convincing, but it is not sufficiently clear what conceptual advance the paper offers beyond the established principle that validation should reflect the intended objective and deployment setting.
Questions for authors

Have the authors manually examined the documents that rank above the designated answer? Knowing how many are genuinely irrelevant could change the assessment of the harm associated with the ranking decline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.