acceptodds
Under review as a conference paper at ICLR 2027

Retrieved but not ranked: surface-form bias in structural retrieval, from mathematics to agent trajectories

Abstract

Embedding retrieval is usually validated where surface form and meaning point the same way. We study the case where they are pulled apart on purpose: retrieving items that share structure but not wording, in two unrelated domains under one protocol, competition mathematics (500 queries against 117,088 items) and embodied-agent trajectories (118 queries against 336). In mathematics, strict Hit@1 at the heaviest disguise tier is 0.0% for both production embedders (bootstrap 95% CI [0.0, 0.0]) while the correct item sits in the top 10 nearly always, and in 95.2 to 99.8% of misses the winner is more lexically similar to the query than the answer. In trajectories, all three embedders fall below hypergeometric chance once gold must differ in object and receptacle: retrieval anchors on literal tokens rather than task structure. A lexical reranker control hurts in mathematics and helps in trajectories (closing 26 to 36% of the recoverable gap), so its sign diagnoses whether a benchmark's surface variation is adversarial or incidental. LLM rerankers recover 5 to 63% of the gap in mathematics and 43 to 76% in trajectories; direction replicates across three judges, but effect sizes, tier profiles, and even which judge is the outlier change with domain, and mathematics gains concentrate on well-known competitions (+19.8 points in one of six cells). Finally, in a paired downstream experiment, oracle retrieval was indistinguishable from adversarially bad retrieval (McNemar p = 0.678), because the solver's 69.5% zero-shot accuracy is largely a truncation proxy, with 97 to 100% accuracy on answers that finish within budget.

Primary area:
datasets and benchmarks
Keywords:
embedding retrieval, reranking, LLM-as-judge, retrieval-augmented generation, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_6Kb5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper benchmarks embedding based retrieval methods, where the documents to be retrieved are rewritten in a semantically equivalent manner (sharing little wording with the original), in the presence of confounding documents that have a different meaning yet share the phrasing structure. Two domains are considered - MathNet retrieval, and ALFWorld trajectories.

Critical strengths
  • Relevant and topical research problem: Determining the impact on retrieval performance with semantically equivalent variants, and similarly phrased, but contradictory variants is practically relevant. The robustness of retrieval and re-ranking methods in such challenging scenarios is useful information for practitioners.
Critical weaknesses
  • Poor writing and presentation: The submission is well below the quality and standards expected at ICLR. The text seems completely automatically generated, with little oversight. The experimental setup is difficult to reconstruct, central concepts such as “structure” and the strict/lenient relevance definitions are not clearly explained, and Sections 3 and 4 give inconsistent descriptions of the mathematical query - target relationship. There are repeated conclusions, and conversational writing, for example “We had hoped a third judge would settle the ordering. It scrambled it instead.”.
  • Limited, incremental contributions: The central claims largely reproduces a lot of the results from MathNet. The additional reranking experiments do not provide any substantially new insights.
  • Sparse discussion of related work: The paper does not position itself in the broader literature, where this problem has been extensively studied.
  • Missing dedicated rerankers: The reranking experiments did not include dedicated rerankers available - this limits the conclusions that can be drawn from these experiments.
Questions for authors
Reviewer_S7AVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyzes the issue of structural retrieval for two domains without nothing in common, mathematics and embodied-agent trajectories, studies the different results of surface-form bias in retrieval and rank of the two domains, and presents some meaningful conclusions.

Critical strengths

The issue and conclusion presented in this paper maybe significant for embedding retrieval research.

Critical weaknesses
  1. The motivation for this research is poorly explained in this paper. There are no examples of surface-form bias provided and it lacks sufficient persuasiveness.
  2. The summary of the contributions (lines 049-071) of this paper is too fragmented and lacks cohesion. Moreover, the content is more like some conclusions.
  3. According to the Abstract, the Related Work section lacks the content of embedding retrieval.
  4. This paper lacks a formal definition of the problem, making it unclear what specific issue it intends to solve.
  5. In Section 6, this paper merely reveals some phenomena or presents some conclusions regarding the embedded retrieval processes in different fields. However, firstly, have these conclusions universality? Secondly, it does not explicitly propose methods for applying these conclusions to solve problems. Therefore, its contribution is insufficient.
  6. The discussion in Section 7 might be useful, but it is not combined with experiments and lacks supporting evidence. Also, how is cross-domain manifested?
  7. The number of references cited in this paper is too small.
  8. The organizational structure of this paper is too loose.
Questions for authors

See Critical Weaknesses

Reviewer_Ps2oOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies structural retrieval on mathematical problems and agent trajectories, comparing embedding retrieval with lexical and LLM reranking. It argues that embeddings favor surface similarity, that the sign of lexical reranking gains reveals how a benchmark was constructed, and that reranking performance varies substantially across judges and domains. A downstream mathematics experiment examines whether retrieval improvements translate into better solver performance. The contribution is primarily an empirical analysis using existing benchmarks and models.

Critical strengths

Using fixed candidate lists across rerankers makes the within-dataset comparisons easier to interpret.

Critical weaknesses
  1. The evaluation does not consistently measure the task given to the models, and the central retrieval claim contradicts the reported results.

The reranking prompts and relevance labels are misaligned. Appendix A asks the model to identify the candidate sharing the same mathematical technique or procedure and explicitly states that exactly one candidate satisfies this criterion. However, Appendix B.1 contains both an easy and a medium equivalent reformulation in the same candidate list, with only one counted as correct. A model can therefore select a mathematically valid match and still be penalized.

The trajectory evaluation has the same problem. The prompt tells the model to ignore object identity, while strict scoring rejects same-object matches even when they have the correct procedure. Appendix B.2 illustrates this directly: both the selected trajectory and the designated gold perform simple placement, but the selected one is rejected because it uses the same object as the query. The additional requirement to select a different object is not stated in the prompt. Since siblings account for approximately 7272–8888% of the reported reranker misses, this mismatch affects a substantial part of the analysis. These errors cannot be straightforwardly interpreted as failures to recognize structure.

The central “retrieved but not ranked” claim is also overstated. The abstract says that the correct item is almost always in the top ten at the hardest tier, whereas Table 1 reports strict Hit@10\mathrm{Hit@10} of only 55.455.4% and 21.021.0%. The 9999–100100% figures in Section 4 use lenient scoring that also counts structurally incorrect near-misses. Retrieving an incorrect decoy does not establish that the correct answer was retrieved. The paper uses this distinction to motivate its main argument but does not maintain it consistently when interpreting the results.

  1. The main explanations are not established by the experiments, and the manuscript’s defensive organization makes its contribution less clear.

The proposed lexical diagnostic is not justified. The sign of a lexical reranker’s gain measures its performance relative to the original embedding ranking on a particular candidate pool. A negative gain can arise simply because lexical reranking discards useful information in the original order; it does not establish that the benchmark’s surface variation is adversarial. The comparison also changes the input representation: in the trajectory domain, embeddings encode the instruction and full trajectory, while the lexical reranker uses only the task description. The paper does not separate the effect of this cleaner input from its proposed explanation of the sign flip.

The downstream conclusion has a further reporting problem. The abstract cites McNemar p=0.678p=0.678 for oracle versus adversarial retrieval, but Section 6 reports that value for no retrieval versus gold retrieval. The stated comparison is therefore not supported by the cited test. Moreover, near-ceiling accuracy on the subset of problems completed under all three conditions does not establish why retrieval failed to improve performance across the full sample.

The standalone Section 10, “What This Paper Does Not Claim,” is particularly poorly conceived. It immediately follows a separate Limitations section and repeats several of the same qualifications. It reads like an internal rebuttal checklist left in the manuscript. The authors should state a clear contribution and keep their conclusions consistent with the evidence throughout the paper. A separate list of claims they disown does not resolve the contradictions between the abstract, evaluation protocol, and reported measurements. Together with the repeated accounts of evaluation incidents, this gives the manuscript the character of an unfinished working report.

These problems affect the central experimental evidence and its interpretation. Addressing them would require revising the evaluation and reconsidering the main conclusions.

Questions for authors
  1. How do the reranking results change when all candidates satisfying the stated same-technique or same-procedure instruction are counted as correct?
  2. Does the lexical improvement in the trajectory domain remain when embeddings and the lexical reranker both use only the task description?
  3. What is the direct paired comparison between oracle and adversarial retrieval? The reported p=0.678p=0.678 concerns no retrieval versus gold retrieval.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.