- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper evaluates LLM creativity as constrained problem solving. On two math benchmarks (OMEGA and Omni-MATH), each problem is paired with versions that ban the reference method, cap the length at 50%, or forbid a character, plus conditions with tool access and threat or reward prompts. A robustness audit swaps numbers and objects on OMEGA, MacGyver and CreativityBench, and the two physical benchmarks are also used for transfer tests. The paper also proposes a proposer-composer harness (PCH), where a small model writes draft solutions for a 32B composer. Accuracy drops as constraints stack, tools help while threats and rewards do not, and PCH improves the composer across benchmarks.
Critical strengths
-
Paired evaluation. Comparing each constrained problem against its own unconstrained baseline with McNemar tests separates task difficulty from the cost of the constraint.
-
Robustness audit. Showing that accuracy swings widely across solution-preserving numeric and object swaps is a useful finding for these benchmarks.
Critical weaknesses
-
The constraint setup does not really match the paper's goal of closing the conventional solving route (L046, L171). On OMEGA, banning a single textbook method leaves gpt-5.5 at 95.5% versus 97.5% at baseline, and length + method still gets 93.5% (Table 2, L1931), although on Omni-MATH the method ban does cost more (62.5% to 37.0%). The drop to 49.0% in the abstract (L022) comes almost entirely from forbidding a character across the whole response (L1232), which is a surface lipogram test that also hits models with visible reasoning hardest (kimi-k2-thinking gets 0.0%, L317-322) while hidden-reasoning models are only checked on their final answer.
-
The large PCH gains can be explained by prompt asymmetry, stronger proposers, and test-set selection rather than creative synergy. The proposer prompt asks the model to show its working (L1346) while the composer prompt does not (L1373), so OLMo-2-32B outputs only about 7 tokens without CoT when run alone (L2465, Table 30). The best math proposer (Phi-4-mini-reasoning) is also much stronger than the composer on its own (pass@1 48.6% vs. 13.9%, Table 26), and averaging over the ten plain proposers instead of taking the test-set best drops accuracy under all three constraints from 59.0% to 21.7% (Tables 6 and 25).
Questions for authors
-
What is the solo accuracy of OLMo-2-32B and Qwen3-32B when given the same "show your working" instruction as the proposer (L1346), and how much PCH gain remains over that baseline?
-
On OMEGA, gpt-5.5 stays at 93.5% under length + method and only drops to 49.0% when the character ban (L1232) is added (Table 2). How does banning a letter in the response measure mathematical route-finding? There is also no human data, and Acar et al. (2019), cited at L305, report an inverted-U where heavy constraints hurt humans too, so what supports the claim at L021 that humans gain under these stacked constraints?
-
Do the model rankings in Table 2 hold if compliance is checked only on the final answer for all models (L317-322), and how does your single-method ban compare to NeoCoder (Lu et al., NAACL 2025), which bans multiple techniques step by step?