- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the generation of cryptic crosswords through a neurosymbolic pipeline. For clue generation, an LLM produces clues and executable DSL derivations, which are checked and revised through a generator–validator loop. For grid construction, the paper encodes crossword rules as satisfiability constraints and explores two ways to incorporate LLM-proposed placements. Experiments compare direct generation with the proposed methods and evaluate clue correctness, answer diversity, several quality proxies, and grid construction performance.
Critical strengths
- The paper addresses an interesting constrained-generation problem. Cryptic crosswords combine semantic definitions, exact letter-level transformations, and global grid constraints. This makes them a useful setting for studying how language generation can be combined with symbolic verification.
- The pipeline provides a concrete integration of generation, validation, and constraint solving. Executable derivations make the proposed wordplay inspectable and support targeted feedback. The experiments demonstrate improvements in weaker models’ clue correctness and show that symbolic methods can construct valid grids where direct LLM generation fails.
Critical weaknesses
- The practical advantage over direct generation remains unclear. GPT-4o improves from 1% to 16% correctness but remains unreliable. For GPT-5.5, correctness stays at 95%, while unique answers increase from 70 to 84 at approximately 2.5 times the reported runtime (Table 4). These results suggest a trade-off rather than a clear overall improvement. A budget-matched comparison of valid, unique outputs is needed to establish the method’s practical value.
- The quality evaluation is not sufficiently validated. The stated dimensions—naturalness, appropriate difficulty, and misdirection—are reasonable, but perplexity and clue–answer semantic distance are not validated against expert judgments of these properties. Although an expert checks correctness, this does not establish broader clue quality. Moreover, Table 5 evaluates only correct clues, leaving some baseline quality estimates based on just one or three examples.
Questions for authors
- Generation efficiency: Under the same budget, how does the full pipeline compare with direct sampling followed by deduplication and equivalent filtering? Could you report the number of valid, unique clues produced per unit of time or cost?
- Validator reliability: Why does correctness remain at 16% and 20% for the weaker models despite the validation loop? Do the remaining errors reflect incorrect validator acceptance or unsuccessful repairs after reaching the iteration limit?
- Quality evaluation: Could you validate perplexity and clue–answer semantic distance against expert judgments of naturalness and misdirection? What evidence supports the assessment of clue difficulty beyond these automatic proxies?
- Grid construction: Does LLM guidance improve upon standalone SAT when the required number of placed answers and computational budget are matched? This would help distinguish efficiency gains from reductions in the number of answers included.
We will answer the questions in order, but we first note that we have updated the presentation of the tables, which can be found here: https://filebin.net/6rol5zg2e10o79w8
Q1: Generation efficiency
It’s true that our pipeline takes more time per correct clue, but we believe this is justified by the number of clues that were repaired (21). It is possible that direct generation would be able to generate more correct clues in the time this repair took, but we note that some clue types are particularly prone to incorrect clues and duplicate clues e.g., homophones, so re-generating in this category is likely to be challenging.
Neither approach currently automates deduplication, so a fair comparison would be to add deduplication into the sampling loop for both direct sampling and our full pipeline. We will add a comparison in the final version to a loop which simply generates in the DSL, discards any invalid or duplicate clues and keeps sampling until a fixed time budget. Note that this is really only possible for generating in the DSL, since there is no accurate way to validate the plain text clues (our experiments with auto-formalization into the proof DSL produced very poor results).
Q2: Validator reliability
These errors are primarily the result of the model failing to repair errors before reaching the iteration limit. We evaluated our evaluator and found it to be 95% accurate on a mix of human-written and generated clues, and will include validator accuracy data in the final version. False positives from the validator are a result of the synonym checker being unduly generous.
Q3: Quality metrics
We have run a user study (with ethics approval), with 46 solvers rating a subset of generated clues in an attempt to provide more accurate quality ratings.
We found that perplexity and semantic distance separate human-written from generated clues at the dataset level (Mann–Whitney p < 0.05, 60 clues per source), but in the user study perplexity did not correlate with perceived quality (Spearman ρ = −0.17, p = 0.50), so we treat these metrics as measures of stylistic resemblance to human clues rather than of quality. Solvers themselves could barely tell the two apart, correctly identifying whether a clue was human-written or generated only 58.5% of the time (Cohen’s κ = 0.17 against a 50% chance rate; 63% for expert solvers). We will update the text accordingly.
However, validity is clearly the most important metric and our pipeline produces more unique valid clues than the baseline.
Q4: grid construction
Both methods use the same binary search over the required number of answers, with a 200 s timeout per level, and our previous table accidentally included only the final solve time for the SAT only baseline. Over the full search, including timed-out levels, the incremental variant places as many answers as SAT (21.0 vs. 21.0, averaged over the six answer sets) using 11% less total time (450 s vs. 508 s), while the soft variant places about 1.8 fewer answers for a smaller saving (468 s). In general, this gain comes from being able to solve problems on which the SAT only case times out. We will clarify in the text
We will include a comparison for specific numbers of words and density requirements in the final version as well as the data on each iteration in the binary search.