acceptodds
Under review as a conference paper at ICLR 2027

LO-Analog: Leakage-Aware Retrieval of Transferable Reasoning Skills for Linguistics Olympiad Problems

Abstract

Retrieving a solved example can help a model reason about a new Linguistics Olympiad problem-but conventional similarity search systematically returns sibling subproblems and shared contexts that expose the target's reasoning path, contaminating evaluation. We introduce LO-Analog, an answer-free benchmark and constrained retrieval framework that exposes and eliminates this leakage. The benchmark indexes 160 Linguini tasks grouped into 73 original-problem and context clusters, represents candidates using answer-free structural signatures, and evaluates relevance, near-leakage, and result-set diversity jointly under a leave-one-task-out protocol. Eight retrieval configurations-spanning BM25, TF-IDF, hybrid scoring, leak filtering, skill supervision, and MMR diversification-are compared on five metrics. The contamination is severe: same-original or same-context items occupy 29.5% of top-five positions in raw rankings. Cluster-level exclusion eliminates every near-leak at both @5 and @10. Under answer-free skill supervision, filtered BM25 improves nDCG@5 from 0.494 to 0.848-a +0.354 gain confirmed by 2,000-resample paired bootstrap (95% interval [0.328, 0.382]). A diversity-aware skill-and-safety ranker simultaneously attains zero near-leak@10, full cluster diversity@10, and nDCG@5 within 0.02 of the oracle upper bound. Ablations show that each filter stage is necessary: context-only filtering still leaks 8.3% at @5, and removing skill supervision collapses nDCG@5 by 0.19. A preregistered, blinded evaluation of 388 unique top-ranked pairs (two annotators plus adjudication) confirms that leakage-aware retrieval improves human-judged analogy helpfulness while sharply reducing human-identified near-leaks; the conclusion survives removal of all public metadata features from the ranking signal. LO-Analog establishes analogy retrieval as a joint relevance-and-safety problem and demonstrates that a substantial fraction of conventional retrieval gains reflects leaked puzzle context rather than genuine structural transfer.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
analogy retrieval, retrieval safety, benchmark leakage, linguistic reasoning, transfer learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_2rUaOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces LO-Analog, a leakage-aware retrieval framework for identifying transferable reasoning skills in Linguistics Olympiad problems without exposing target-specific information. The authors construct an answer-free benchmark of 160 tasks and evaluate retrieval methods combining similarity ranking, leakage filtering, and diversification. Experiments show that leakage filtering substantially reduces contamination while improving retrieval relevance, human-judged helpfulness, and downstream LLM reasoning performance.

Critical strengths
  1. The paper offers an interesting perspective on analogy retrieval by considering information leakage alongside retrieval relevance, which could be meaningful for real-world applications.
  2. The paper provides extensive experiments, including comparisons with multiple baselines, ablation studies, human evaluations, and downstream LLM experiments.
Critical weaknesses
  1. Insufficient discussion and positioning of related work. The paper lacks sufficient discussion of prior literature to establish a clear research background. The introduction briefly mentions existing benchmarks and retrieval approaches without adequately explaining their limitations or motivation for this work. Moreover, the related work section mainly lists relevant research directions without discussing their connections to the proposed approach. Consequently, it is difficult to understand how this work is positioned within the existing literature.
  2. Insufficient justification of the framework design. The proposed framework combines existing retrieval methods, rule-based leakage filtering, and MMR diversification, but does not sufficiently explain how these components enable the retrieval of transferable reasoning skills beyond superficial similarity matching.
  3. Unclear methodological novelty. While the framework addresses leakage through explicit exclusion rules, it remains unclear whether the contribution lies in a novel retrieval methodology or primarily in applying existing techniques to a newly defined retrieval setting.
  4. Poor presentation of experimental results. The experimental section contains numerous short subsections, making the presentation fragmented and the main findings difficult to follow. Using figures and tables could help readers compare results and identify key insights.
Questions for authors
  1. Do the authors have any plan to publicly release the new benchmark? Based on my understanding, you adopted the original Linguini dataset. Adding a datacard will help readers understand how you processed the original data.
  2. The reasoning benchmark is one of the core contributions of this paper. Besides the experimental results in the paper which mainly focused on retrieval, how could this benchmark make more impact in the AI community?
  3. For the hybrid safety ranker, how did the authors decide the weights for each term?
  4. For the models used in "secondary solver evidence", could you share more details beyond the model size? Are they some LLM APIs or fine-tuned open-sourced model?
Reviewer_7sQZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies retrieval-induced evaluation leakage in Linguistics Olympiad tasks. In this setting, sibling subproblems can share the same original puzzle and linguistic context, making them highly attractive to similarity-based retrieval while potentially exposing target-specific information. LO-Analog constructs an answer-free index of 160 Linguini tasks organized into 73 original-problem/context clusters, excludes unsafe candidates, and compares sparse, hybrid, and diversified retrieval. It evaluates retrieval quality through automatic metrics and blinded human judgments, with downstream solver experiments as secondary evidence. The intervention concerns retrieval at inference time, rather than preventing contamination of model-training data.

Critical strengths

The paper is clean with respect to the problem it studies. It identifies a genuine evaluation concern in this dataset structure: merely excluding the exact target record is insufficient when other records contain closely related parts of the same original puzzle. Making this dependency explicit is useful for researchers working with such benchmarks.

The evaluation goes beyond an automatic demonstration that a filter removes items matching its own rules. In the blinded human study, SafeHelpful@1 increases from 7.5% for raw BM25 to 61.3% for filtered BM25 and 65.6% for hybrid MMR. The manuscript also reports uncertainty and acknowledges that the incremental top-1 benefit of hybrid MMR over filtering alone is not statistically established. This is a positive aspect of the reporting.

The counterfactual study is also useful. It attempts to distinguish exposure to sibling-puzzle content from genuine use of a safe analogy, rather than treating improved solver accuracy as sufficient evidence of reasoning transfer.

Critical weaknesses

Main: the retrieval-augmented evaluation setting is one the authors introduced themselves and then solved themselves, so the contribution is limited.

While the paper demonstrates substantial sibling-problem leakage under its constructed retrieval protocol, it does not establish that this issue affects the standard evaluation of Linguini or other widely adopted benchmarks. The proposed mitigation largely follows established provenance-filtering practices. Consequently, the broader practical significance and incremental contribution remain limited.

Secondary: the method is too simple.

Having seen this problem, I would have arrived at roughly the same idea as the paper even without reading it, and I suspect an agent could do the same. Modern data-cleaning pipelines already perform this kind of AI-based review.

Questions for authors
  1. Can you summarize whether any existing benchmarks are affected by a similar failure mode? If you could correct the published numbers for public benchmarks, that would count as a contribution.

Answering this question alone is unlikely to raise my score; I would also want to see the other reviewers’ questions and your discussion of whether the paper’s motivation holds.

Reviewer_BuxuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper describe a new "analogy retrieval" pipeline for Linguistics Olympiad (LO) problems. The proposed retrieval pipeline filters out all kind of leaked solutions from retrieved content, including structurally similarly task, semantically closed solutions etc. This is a good engineering report which can be useful to other AI reasoning domains (math, code etc)

Critical strengths

This is a good engineering report which can be useful to other AI reasoning domains (math, code etc)

Critical weaknesses

This is a good tech report, but it has very minor novelty from scientific point of view.

Questions for authors

Can you clarify "skill supervision" part of proposed pipeline, please?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.