Retrieved but not ranked: surface-form bias in structural retrieval, from mathematics to agent trajectories

Embedding retrieval is usually validated where surface form and meaning point the same way. We study the case where they are pulled apart on purpose: retrieving items that share structure but not wording, in two unrelated domains under one protocol, competition mathematics (500 queries against 117,088 items) and embodied-agent trajectories (118 queries against 336). In mathematics, strict Hit@1 at the heaviest disguise tier is 0.0% for both production embedders (bootstrap 95% CI [0.0, 0.0]) while the correct item sits in the top 10 nearly always, and in 95.2 to 99.8% of misses the winner is more lexically similar to the query than the answer. In trajectories, all three embedders fall below hypergeometric chance once gold must differ in object and receptacle: retrieval anchors on literal tokens rather than task structure. A lexical reranker control hurts in mathematics and helps in trajectories (closing 26 to 36% of the recoverable gap), so its sign diagnoses whether a benchmark's surface variation is adversarial or incidental. LLM rerankers recover 5 to 63% of the gap in mathematics and 43 to 76% in trajectories; direction replicates across three judges, but effect sizes, tier profiles, and even which judge is the outlier change with domain, and mathematics gains concentrate on well-known competitions (+19.8 points in one of six cells). Finally, in a paired downstream experiment, oracle retrieval was indistinguishable from adversarially bad retrieval (McNemar p = 0.678), because the solver's 69.5% zero-shot accuracy is largely a truncation proxy, with 97 to 100% accuracy on answers that finish within budget.