- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
Cusp evaluates finite-set constraint reasoning through assignment recovery and solution-space classification. It generates constraints from sampled sets, prunes them while checking uniqueness, and distinguishes a retention sweep from derived deletion-tested minimal cores. Evaluations vary constraint structure, universe size, inference settings, and prompt framing. The strongest controlled prompt result is a cue-by-answer-space interaction: asserting uniqueness changes set accuracy by -0.4 percentage points when an assignment is required and +7.8 points when a non-unique response is permitted. The primary classification result excludes the surface-detectable unsatisfiable class and reaches 78.6% on UNIQUE versus MULTIPLE. The benchmark offers useful tests of set reasoning, but its added value over existing benchmarks, relevance to practical tasks, and explanations of model behavior need stronger support.
Critical strengths
- Cusp tests whether models can recover sets and judge whether a solution is unique. Its solver-verified instances and deletion-tested minimal cores extend the set-operation tasks in SetLexSem and the CSP puzzles in ZebraLogic.
- The paper checks earlier labeling errors, compares prompts on the same instances, and tests added information against repetition and padding. It also examines reasoning budgets and separates the harder uniqueness decision from easy contradiction detection.
Critical weaknesses
- The contribution mainly combines ideas in synthetic constraint benchmarks, solver verification, and prompt variation. Methodological novelty is limited, and the paper does not sufficiently establish what useful diagnosis this combination adds beyond benchmarks such as SetLexSem and ZebraLogic.
- Cusp measures single-shot reasoning over synthetic set constraints, but its relevance to practical mathematical work or general reasoning has not been established, the benchmark’s practical significance remains uncertain.
- The paper attributes prompt effects to whether models commit to an answer and interprets scaling failures as evidence about their solving strategies. But changes in output accuracy and refusal rates do not establish these internal mechanisms or rule out other explanations.
- All evaluated systems are accessed through APIs, including the open-weight models. No locally deployed checkpoint is evaluated. This may cast doubt on the reproducibility of this paper.
- The presentation is overly defensive, using repeated caveats, corrections to earlier interpretations, and future plans that interrupt the main argument explanation. Also, too much space is spent explaining the limits.
- The reported reversal between GPT-5.4 and o4-mini after adjusting reasoning effort shows that these rankings depend on configuration. Therefore, accuracy should be compared alongside token use and output limits.
Questions for authors
- I suggest checking whether the low-retention and uniqueness errors persist when models can run code to verify their answers. A representative subset would help assess the relevance of these failures to practical use.
- With base instances and inference settings held fixed, what additional failure patterns do deletion-tested minimal cores reveal compared with the retention sweep?
- For the -0.4 and +7.8 percentage-point cue effects in Section 5, please report how individual instances move between correct assignments, wrong assignments, and non-unique responses. These transitions would help assess the claim that computation stays unchanged.
- Section 5 would be clearer with the matched-instance comparison as its main result and the repeated qualifications consolidated around it, since the original Implicit/Explicit comparison also changes the constraint regime.
4 comments
Continue to the answer to Q2:
Pattern 2. Spurious refusal. When the prompt offers NO_UNIQUE_SOLUTION, models take it on instances that have exactly one solution, sometimes after the trace has already derived that solution. Example, :
| # | Constraint |
|---|---|
| 1 | |
| 2 | |
| 3 | |
| 4 |
Ground truth: , , . Opus 4.6's trace:
| Step | Reasoning trace | Status |
|---|---|---|
| 1 | and ; and ; so | correct |
| 2 | , so | correct |
| 3 | " contains and possibly some of " | wrong: , so would put into , violating c.3; exactly, and then forces |
| 4 | ", , works" | the correct answer, stated in full |
| 5 | ", , also works" | violates c.3 () |
| 6 | <solution>NO_UNIQUE_SOLUTION</solution> | spurious refusal |
Refusals on the 891 cores (Explicit prompt, neutral cue) whose trace had already stated every set correctly:
| Model | Refusals | Derived all sets, then refused | % of refusals |
|---|---|---|---|
| Opus 4.6 | 155 | 28 | 18.1% |
| Sonnet 4.6 | 225 | 50 | 22.2% |
| Sonnet 4.5 | 38 | 11 | 28.9% |
| Sonnet 4 | 95 | 13 | 13.7% |
| Haiku 4.5 | 124 | 15 | 12.1% |
| Gemini 2.5 Flash | 51 | 8 | 15.7% |
| Gemini 2.5 Flash-Lite | 6 | 0 | 0.0% |
| Mistral Large | 661 | 16 | 2.4% |
| Qwen3 32B | 54 | 1 | 1.9% |
| Llama 3.3 70B | 732 | 10 | 1.4% |
| Llama 4 Maverick | 164 | 12 | 7.3% |
The sweep cannot show this: with about 59 constraints a redundant value constraint removes the doubt, and under the Implicit prompt the sweep traces voice uniqueness doubt on 0–1 of 891 instances for the Anthropic models.
Pattern 3. Sensitivity to which core is shown. The same hidden answer has many exactly-minimal cores, and which one the prune order leaves decides the outcome. Illustration for the answer , , (36 true constraints):
| Core 1 (App. A.2) | Core 2 (illustrative, solver-verified unique and minimal) | |
|---|---|---|
| Constraints | ; ; ; | ; ; ; |
| Unique / minimal | yes / yes | yes / yes |
| Solving needs | composition (guess-and-check) | reading only (disjoint ⇒ ) |
| Opus 4.6 | wrong | right |
In the prune-order study (100 bases × 10 independently pruned cores) model rankings are identical across orders (Spearman 1.00) while single-instance outcomes flip (accuracy ICC 0.06–0.16). Restricting to bases whose sweep instance the model solved:
| Model | Sweep solved (bases) | Cores all right | Cores mixed | Cores all wrong | Mean cores right / 10 |
|---|---|---|---|---|---|
| Opus 4.6 | 17 | 8 | 9 | 0 | 9.1 |
| Haiku 4.5 | 15 | 0 | 15 | 0 | 5.1 |
| Qwen3 32B | 8 | 0 | 8 | 0 | 3.5 |
Within those mixed bases we asked whether the failing cores are measurably harder than the passing cores of the same base (compared with the passing cores' median): readability = share of sets propagation alone fixes; value share = share of value constraints; open decisions = element–set decisions no single constraint fixes.
| Model | Bases | Wrong cores | Readability: harder / same / easier | Value share | Open decisions | Failure types (composition / reading / no answer) |
|---|---|---|---|---|---|---|
| Opus 4.6 | 11 | 20 | 40% / 25% / 35% | 45% / 25% / 30% | 55% / 5% / 40% | 18 / 1 / 1 |
| Haiku 4.5 | 19 | 102 | 63% / 16% / 22% | 49% / 23% / 28% | 56% / 4% / 40% | 72 / 22 / 8 |
| Qwen3 32B | 12 | 80 | 39% / 25% / 36% | 55% / 22% / 22% | 38% / 4% / 59% | 38 / 19 / 23 |
The failing cores are not systematically harder on any of the three measures (within-base for Opus and Haiku), and 90% of Opus's failures on them are composition failures. The outcome depends on which minimal subset the model sees, not on a measurable difficulty of that subset. This brittleness is itself a boundary finding and the reason the paper restricts itself to aggregate claims on the cores.
Continue to question 3 and 4:
Q3: We thank the reviewer for asking this question, as the data support our argument that pragmatics are relevant to the evaluation of language models on reasoning tasks. As shown in the transition probability matrix below, the presence of the cue that the problem is uniquely solvable results in 35% of previously-wrong responses to become correct, and 71% of previously-refused responses becoming correct. The statistically significant changes in accuracy are due to the previous refusals becoming either correct or wrong are shown in bold.
| neutral cue → asserted cue | CORRECT | WRONG | REFUSE |
|---|---|---|---|
| CORRECT (5,462) | 4,932 (90.3%) | 455 (8.3%) | 75 (1.4%) |
| WRONG (1,364) | 479 (35.1%) | 874 (64.1%) | 11 (0.8%) |
| REFUSE (910) | 648 (71.2%) | 213 (23.4%) | 49 (5.4%) |
Percentages are calculated within rows; counts are pooled over nine models.
- Significant at p<0.001 across three pairwise likelihood-ratio tests comparing each transition with its reverse (df=1). CORRECT ↔ REFUSE: G²=520.47; WRONG ↔ REFUSE: G²=222.78. The unmarked off-diagonal pair is not significant (CORRECT ↔ WRONG: G²=0.62, p=0.43)
Q4 (structure of Section 5). We agree and will restructure Section 5 as you suggest: the matched-instance factorial becomes the main result, the Implicit/Explicit aggregate becomes the motivating observation with its regime difference stated once, and the qualifications are consolidated around it.
Every new number above was computed from artifacts that are already in the public release, without any new model runs. With Section 5 restructured as you propose, we would be grateful if you would reconsider your assessment.
Dear Reviewer BhZy,
Thanks for your time and effort in reviewing our paper. As the author-reviewer discussion period will end in 1 day, we gently request that you read the rebuttal and check if your concerns are addressed. Thanks again for your time and help in improving the quality of our paper.
Regards from authors.
We thank the reviewer for engaging with the paper down to the appendices. We acknowledge the point on presentation: the repeated caveats, corrections of earlier interpretations and etc. The next revision will state each qualification in one place, and keep the main text on the results. Below is answer to each of your question:
Q1 (code execution). We ran this check: when a model writes a z3 program instead of answering in context, the frontier models recover every core they had failed (Sonnet 4.6 25/25, Opus 4.6 183/183, all constraints encoded correctly), Haiku 4.5 recovers 84% (359/427) and Llama 3.3 70B 52% (398/760), the remainder being program faults (crashes, syntax errors, mis-encoded constraints) rather than reasoning errors, so the low-retention and uniqueness failures measure in-context composition and commitment, which a solver call removes. This complements the global response, where Z3 search statistics (decisions, conflicts, propagations) explain almost none of the variance in model accuracy (R2=0.05) while what the constraints determine explains half of it (R2=0.50): what is hard for a complete solver is not what is hard for these models, and handing the models the solver removes exactly the step they fail.
Q2 With base instances, model and inference settings fixed, the cores expose three failure patterns that the sweep does not.
Pattern 1. The failure moves from reading a constraint to inference by composition. We score a wrong answer as a reading failure when it misplaces an element that a single constraint already fixes (the first-order must/cannot rules), and as a composition failure when every single-constraint inference is right and the error lies only on elements that two or more constraints determine jointly.
Example (App. A.2), U=0,1,2,4,7,10,13,21,25,29,33,34:
On the sweep instance of comparable size (App. A.3) every set is read off two or three value constraints and all are correct. A no-search propagation baseline fully solves 33.6% of the sweep but 2.6% of the cores; on the cores, must/cannot accuracy stays above 92% and second-order accuracy at 98–100% while can elements fall to 86–87% for mid-tier models; 98.8% of wrongly included elements belong to another target set.
Wrong answers on the 891 cores (Implicit prompt), by failure type:
What the cores add, same bases and same prompt:
The added composition failures almost all violate a value constraint
The pattern is guess-and-check with the check skipped: a cardinality should decide the open elements, the model resolves them by guessing, and commits without re-verifying an explicit value constraint the guess breaks.