- The evaluation does not consistently measure the task given to the models, and the central retrieval claim contradicts the reported results.
The reranking prompts and relevance labels are misaligned. Appendix A asks the model to identify the candidate sharing the same mathematical technique or procedure and explicitly states that exactly one candidate satisfies this criterion. However, Appendix B.1 contains both an easy and a medium equivalent reformulation in the same candidate list, with only one counted as correct. A model can therefore select a mathematically valid match and still be penalized.
The trajectory evaluation has the same problem. The prompt tells the model to ignore object identity, while strict scoring rejects same-object matches even when they have the correct procedure. Appendix B.2 illustrates this directly: both the selected trajectory and the designated gold perform simple placement, but the selected one is rejected because it uses the same object as the query. The additional requirement to select a different object is not stated in the prompt. Since siblings account for approximately 72–88 of the reported reranker misses, this mismatch affects a substantial part of the analysis. These errors cannot be straightforwardly interpreted as failures to recognize structure.
The central “retrieved but not ranked” claim is also overstated. The abstract says that the correct item is almost always in the top ten at the hardest tier, whereas Table 1 reports strict Hit@10 of only 55.4 and 21.0. The 99–100 figures in Section 4 use lenient scoring that also counts structurally incorrect near-misses. Retrieving an incorrect decoy does not establish that the correct answer was retrieved. The paper uses this distinction to motivate its main argument but does not maintain it consistently when interpreting the results.
- The main explanations are not established by the experiments, and the manuscript’s defensive organization makes its contribution less clear.
The proposed lexical diagnostic is not justified. The sign of a lexical reranker’s gain measures its performance relative to the original embedding ranking on a particular candidate pool. A negative gain can arise simply because lexical reranking discards useful information in the original order; it does not establish that the benchmark’s surface variation is adversarial. The comparison also changes the input representation: in the trajectory domain, embeddings encode the instruction and full trajectory, while the lexical reranker uses only the task description. The paper does not separate the effect of this cleaner input from its proposed explanation of the sign flip.
The downstream conclusion has a further reporting problem. The abstract cites McNemar p=0.678 for oracle versus adversarial retrieval, but Section 6 reports that value for no retrieval versus gold retrieval. The stated comparison is therefore not supported by the cited test. Moreover, near-ceiling accuracy on the subset of problems completed under all three conditions does not establish why retrieval failed to improve performance across the full sample.
The standalone Section 10, “What This Paper Does Not Claim,” is particularly poorly conceived. It immediately follows a separate Limitations section and repeats several of the same qualifications. It reads like an internal rebuttal checklist left in the manuscript. The authors should state a clear contribution and keep their conclusions consistent with the evidence throughout the paper. A separate list of claims they disown does not resolve the contradictions between the abstract, evaluation protocol, and reported measurements. Together with the repeated accounts of evaluation incidents, this gives the manuscript the character of an unfinished working report.
These problems affect the central experimental evidence and its interpretation. Addressing them would require revising the evaluation and reconsidering the main conclusions.