- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces LO-Analog, a leakage-aware retrieval framework for identifying transferable reasoning skills in Linguistics Olympiad problems without exposing target-specific information. The authors construct an answer-free benchmark of 160 tasks and evaluate retrieval methods combining similarity ranking, leakage filtering, and diversification. Experiments show that leakage filtering substantially reduces contamination while improving retrieval relevance, human-judged helpfulness, and downstream LLM reasoning performance.
Critical strengths
- The paper offers an interesting perspective on analogy retrieval by considering information leakage alongside retrieval relevance, which could be meaningful for real-world applications.
- The paper provides extensive experiments, including comparisons with multiple baselines, ablation studies, human evaluations, and downstream LLM experiments.
Critical weaknesses
- Insufficient discussion and positioning of related work. The paper lacks sufficient discussion of prior literature to establish a clear research background. The introduction briefly mentions existing benchmarks and retrieval approaches without adequately explaining their limitations or motivation for this work. Moreover, the related work section mainly lists relevant research directions without discussing their connections to the proposed approach. Consequently, it is difficult to understand how this work is positioned within the existing literature.
- Insufficient justification of the framework design. The proposed framework combines existing retrieval methods, rule-based leakage filtering, and MMR diversification, but does not sufficiently explain how these components enable the retrieval of transferable reasoning skills beyond superficial similarity matching.
- Unclear methodological novelty. While the framework addresses leakage through explicit exclusion rules, it remains unclear whether the contribution lies in a novel retrieval methodology or primarily in applying existing techniques to a newly defined retrieval setting.
- Poor presentation of experimental results. The experimental section contains numerous short subsections, making the presentation fragmented and the main findings difficult to follow. Using figures and tables could help readers compare results and identify key insights.
Questions for authors
- Do the authors have any plan to publicly release the new benchmark? Based on my understanding, you adopted the original Linguini dataset. Adding a datacard will help readers understand how you processed the original data.
- The reasoning benchmark is one of the core contributions of this paper. Besides the experimental results in the paper which mainly focused on retrieval, how could this benchmark make more impact in the AI community?
- For the hybrid safety ranker, how did the authors decide the weights for each term?
- For the models used in "secondary solver evidence", could you share more details beyond the model size? Are they some LLM APIs or fine-tuned open-sourced model?