acceptodds
Under review as a conference paper at ICLR 2027

PolyCross: A Benchmark for Identifying Hidden Meanings in Short Phrases with LLMs

Abstract

Identifying hidden meanings in short text snippets is a challenging task with applications in legal documents, safety-critical technical documentation, creative writing, and word puzzles. In this work we investigate the capacity of large language models (LLMs) to identify concealed meanings in short phrases, provide rationales for the existence of meanings identified, and translate phrases with multiple meanings into another language. For the empirical evaluation we introduce POLYCROSS, a benchmark dataset built from a special class of crosswords clues in Romanian which we call polysemantic clues. Each such clue carries multiple meanings, including a subtle meaning that leads to the actual crosswords solution. The baseline is THEMCROSS, a dataset of general-knowledge Romanian crosswords clues with no hidden meanings. We use the two data sets to evaluate three LLMs (GPT-OSS, LLaMA and Granite) with progressively stronger hints about solutions. We assess LLM accuracy of finding crosswords solutions and the quality of LLM rationales to explain their solutions. Our results show that the performance of modern LLMs is substantially lower on polysemantic clues (POLYCROSS) than on the baseline set (THEMCROSS). This highlights an important gap in current LLM capabilities and establishes POLYCROSS as a benchmark to measure future progress on this problem.

Primary area:
datasets and benchmarks
Keywords:
language models, crossword puzzles, polysemantic clues

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_joEKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduce PolyCross, a benchmark of around 26k Romanian crossword clues where each clue have hidden second meaning that lead to the answer. As control, authors use ThemCross, around 1.1k general-knowledge clues from thematic puzzles. They evaluate five LLMs (Granite-4.0-small, Llama-3.3-70B, gpt-oss-120B, Claude Opus 4.8, LLMic-v2-3B) with increasing hints (answer length, then 1–4 prefix letters), CoT, and a translate-to-English prompt. A single expert also score the rationales of three models on 0–5 scale. Main finding is LLMs perform much worse on PolyCross than ThemCross.

Critical strengths
  1. New dataset in low-resource language with natural ambiguity, and it is really hard. Without prefix hint, even Claude Opus 4.8 is below 10%.

  2. Table 2 stratification by number of hidden letters is good analysis. It separate hint leakage from actual solving, and the error categories in Fig. 2 also helpful.

Critical weaknesses
  1. Not clear if the task is well-defined without the crossword grid. There is no human baseline, and in real crossword people have crossing letters. Table 3 even show the same clue "Rezultatul adunarii" with two different gold answers (ATOT and NOIAN), so one clue can have more than one valid answer and exact match will underestimate. Many gold answers are also rare, inflected, or multi-word (INELDEAUR, ARTAG, VINCUSIFON). Also ThemCross is not clean control, because it differ in source, size, and question type (trivia vs wordplay), not only polysemy. So I think the gap cannot be fully attributed to hidden meaning.

  2. The evaluation of rationale and translation is weak. Rationale is scored by one self-selected expert, without agreement measure, without sample size, only for three models, and the hint level is not stated. Since score is only on correct answers, low PolyCross score may come from answer that is solved by prefix and then rationalized. For translation, there is no evaluation of translation quality at all. The claim that v5 is "inferior" also look unfair: with the same length hint as v2, v5 is equal or better for most models (e.g., Claude on ThemCross 65.1 → 73.5). Headline numbers like 77% and 86% mostly come from hint, since 44% of answers is fully revealed at k=4.

Questions for authors

see weakness

Reviewer_jFppOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PolyCross, a Romanian crossword benchmark containing 26,184 clue-answer pairs designed to evaluate LLMs' understanding of hidden meanings. It also constructs ThemCross, a collection of conventional crossword clues used as a comparison set. Several LLMs are evaluated under different hinting strategies, including answer length, prefixes, chain-of-thought prompting, and translation-based reasoning. The experiments show substantially lower accuracy and rationale quality on PolyCross than on ThemCross.

Critical strengths

The paper introduces a relatively large collection of Romanian crossword clues with non-obvious interpretations. This provides a potentially useful resource for studying lexical ambiguity and wordplay.

The experiments consider several models, different hint levels, error categories, and human evaluation of rationales. The analysis of fully revealed answers is particularly useful for interpreting the results.

Critical weaknesses
  1. The comparison between PolyCross and ThemCross is potentially confounded. The datasets originate from different types of crossword puzzles and may differ in vocabulary, cultural knowledge, clue complexity, and answer distributions. Lower accuracy cannot be attributed solely to polysemy.

  2. The hinting protocol substantially reduces the difficulty of the task. At the four-letter hint level, approximately 44% of answers are fully revealed.

  3. The rationale evaluation relies on a single expert and includes only instances where the predicted answer is correct. The lack of independent agreement measures and the restricted evaluation subset limit the reliability of conclusions about explanation quality.

  4. The translation experiment does not directly measure whether multiple meanings are preserved. It evaluates crossword-solving accuracy after translation, which conflates translation quality, semantic preservation, and answer inference.

  5. The benchmark does not sufficiently isolate hidden-meaning understanding from crossword-solving ability. The dataset primarily provides clue-answer pairs without explicit annotations of the obvious and hidden interpretations, making it difficult to determine what specific semantic ability is being evaluated.

Questions for authors
  1. Clarify how the benchmark isolates hidden-meaning understanding from general crossword-solving difficulty.

  2. Provide stronger evidence for the comparability of PolyCross and ThemCross.

  3. Further distinguish semantic reasoning gains from answer-prefix information.

  4. Clarify the reliability of rationale evaluation and the validity of translation-based assessment.

Reviewer_uaXPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces POLYCROSS, a novel benchmark dataset designed to evaluate the capability of Large Language Models (LLMs) to identify hidden, polysemantic meanings in short phrases. Built from specialized crossword clues in Romanian, POLYCROSS challenges models to look beyond obvious, misleading interpretations and discover subtle meanings that lead to the correct solutions. The authors also provide a control dataset, THEMCROSS, consisting of monosemantic general-knowledge crossword clues. The empirical results on several LLMs demonstrate that all evaluated models struggle significantly more with POLYCROSS than with the THEMCROSS baseline, highlighting a substantial gap in current LLMs' deep semantic disambiguation and reasoning capabilities.

Critical strengths
  1. While existing benchmarks focus on trivia, straightforward question-answering, or structured cryptic crosswords, POLYCROSS targets natural language polysemy and contextual ambiguity, serving as a rigorous testbed for advanced semantic understanding.

  2. The benchmark is constructed from a large corpus of over 26,000 cleaned records and is paired with a well-matched monosemantic baseline dataset (THEMCROSS) from the same domain, ensuring that performance differences stem from semantic complexity rather than domain mismatch.

Critical weaknesses
  1. The dataset and evaluation are currently restricted to Romanian. While interesting, this limits the immediate cross-lingual and global generalizability of the findings to other major linguistic families.

  2. As shown in the error analysis, smaller or less capable models frequently fail due to formatting or structural violations (e.g., wrong answer lengths or prefix mismatches) rather than purely conceptual misunderstandings.

  3. The paper notes that multi-meaning clues are extremely hard to translate, but it treats this mostly as an inherent limitation of the task.How to mitigate the gap?

  4. A lexical frequency analysis is necessary on the failure cases to disentangle whether errors are driven by rare vocabulary or genuine reasoning failures.

Questions for authors

Please see the Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.