- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduce PolyCross, a benchmark of around 26k Romanian crossword clues where each clue have hidden second meaning that lead to the answer. As control, authors use ThemCross, around 1.1k general-knowledge clues from thematic puzzles. They evaluate five LLMs (Granite-4.0-small, Llama-3.3-70B, gpt-oss-120B, Claude Opus 4.8, LLMic-v2-3B) with increasing hints (answer length, then 1–4 prefix letters), CoT, and a translate-to-English prompt. A single expert also score the rationales of three models on 0–5 scale. Main finding is LLMs perform much worse on PolyCross than ThemCross.
Critical strengths
-
New dataset in low-resource language with natural ambiguity, and it is really hard. Without prefix hint, even Claude Opus 4.8 is below 10%.
-
Table 2 stratification by number of hidden letters is good analysis. It separate hint leakage from actual solving, and the error categories in Fig. 2 also helpful.
Critical weaknesses
-
Not clear if the task is well-defined without the crossword grid. There is no human baseline, and in real crossword people have crossing letters. Table 3 even show the same clue "Rezultatul adunarii" with two different gold answers (ATOT and NOIAN), so one clue can have more than one valid answer and exact match will underestimate. Many gold answers are also rare, inflected, or multi-word (INELDEAUR, ARTAG, VINCUSIFON). Also ThemCross is not clean control, because it differ in source, size, and question type (trivia vs wordplay), not only polysemy. So I think the gap cannot be fully attributed to hidden meaning.
-
The evaluation of rationale and translation is weak. Rationale is scored by one self-selected expert, without agreement measure, without sample size, only for three models, and the hint level is not stated. Since score is only on correct answers, low PolyCross score may come from answer that is solved by prefix and then rationalized. For translation, there is no evaluation of translation quality at all. The claim that v5 is "inferior" also look unfair: with the same length hint as v2, v5 is equal or better for most models (e.g., Claude on ThemCross 65.1 → 73.5). Headline numbers like 77% and 86% mostly come from hint, since 44% of answers is fully revealed at k=4.
Questions for authors
see weakness