acceptodds
Under review as a conference paper at ICLR 2027

Getting It Right, Under Constraint: Evaluating and Improving LLM Creativity

11 views

Abstract

One psychological account defines creativity as *constrained problem solving*, shaped by five situational factors: pressure, pleasure, opportunity, prototypes (earlier works to imitate and transform), and helpers. Current creativity benchmarks for large language models (LLMs) vary none of these factors. Their evaluation either scores correctness with the conventional route open or closes it without comparing against an unconstrained counterpart, and we find it is unstable: swapping incidental numbers alone costs models 8.9 to 15.2 points on OMEGA. We therefore build OMEGA-Constraint and Omni-MATH-Constraint, pairing each problem with versions that ban the reference method, cap length, or forbid a character, closing the conventional route while another stays open. We then add tool access as opportunity, threats as pressure, and rewards as pleasure. Whereas constraint can sharpen human creativity, LLMs lose accuracy as constraints accumulate (under all three, gpt-5.5 drops from 97.5 to 49.0 percent on OMEGA) and reapply banned methods under new names. As in humans, tools help every model, but threats and rewards of up to a million dollars move none by more than 6 points. Our proposer-composer harness, based on stage theories of creativity, supplies the remaining two factors: a small proposer (the helper) brainstorms candidates (the prototypes), which a larger composer develops into its own answer. With the best of ten proposers, both 32B composers gain in every condition, and a single 3.8B proposer lifts gpt-4.1 from 9.0 to 25.0 percent under all three constraints on OMEGA. On constrained math, our harness with a String Seed of Thought-prompted proposer beats plain and fine-tuned proposers in 17 of 28 conditions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, creativity, evaluation, constrained problem solving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_BPiWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates LLM creativity as constrained problem solving. On two math benchmarks (OMEGA and Omni-MATH), each problem is paired with versions that ban the reference method, cap the length at 50%, or forbid a character, plus conditions with tool access and threat or reward prompts. A robustness audit swaps numbers and objects on OMEGA, MacGyver and CreativityBench, and the two physical benchmarks are also used for transfer tests. The paper also proposes a proposer-composer harness (PCH), where a small model writes draft solutions for a 32B composer. Accuracy drops as constraints stack, tools help while threats and rewards do not, and PCH improves the composer across benchmarks.

Critical strengths
  1. Paired evaluation. Comparing each constrained problem against its own unconstrained baseline with McNemar tests separates task difficulty from the cost of the constraint.

  2. Robustness audit. Showing that accuracy swings widely across solution-preserving numeric and object swaps is a useful finding for these benchmarks.

Critical weaknesses
  1. The constraint setup does not really match the paper's goal of closing the conventional solving route (L046, L171). On OMEGA, banning a single textbook method leaves gpt-5.5 at 95.5% versus 97.5% at baseline, and length + method still gets 93.5% (Table 2, L1931), although on Omni-MATH the method ban does cost more (62.5% to 37.0%). The drop to 49.0% in the abstract (L022) comes almost entirely from forbidding a character across the whole response (L1232), which is a surface lipogram test that also hits models with visible reasoning hardest (kimi-k2-thinking gets 0.0%, L317-322) while hidden-reasoning models are only checked on their final answer.

  2. The large PCH gains can be explained by prompt asymmetry, stronger proposers, and test-set selection rather than creative synergy. The proposer prompt asks the model to show its working (L1346) while the composer prompt does not (L1373), so OLMo-2-32B outputs only about 7 tokens without CoT when run alone (L2465, Table 30). The best math proposer (Phi-4-mini-reasoning) is also much stronger than the composer on its own (pass@1 48.6% vs. 13.9%, Table 26), and averaging over the ten plain proposers instead of taking the test-set best drops accuracy under all three constraints from 59.0% to 21.7% (Tables 6 and 25).

Questions for authors
  1. What is the solo accuracy of OLMo-2-32B and Qwen3-32B when given the same "show your working" instruction as the proposer (L1346), and how much PCH gain remains over that baseline?

  2. On OMEGA, gpt-5.5 stays at 93.5% under length + method and only drops to 49.0% when the character ban (L1232) is added (Table 2). How does banning a letter in the response measure mathematical route-finding? There is also no human data, and Acar et al. (2019), cited at L305, report an inverted-U where heavy constraints hurt humans too, so what supports the claim at L021 that humans gain under these stacked constraints?

  3. Do the model rankings in Table 2 hold if compliance is checked only on the final answer for all models (L317-322), and how does your single-method ban compare to NeoCoder (Lu et al., NAACL 2025), which bans multiple techniques step by step?

Reviewer_VdPpOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors define creativity as constrained problem solving. They build constrained versions of OMEGA and Omni-MATH and they vary tools, threats, and rewards in the prompts. They then propose a "proposer-composer" harness, where small models "brainstorm" and a larger model writes the answer. Overall, not sure what to grab from this paper. Table 2 is a bit too dense for a table.

Critical strengths

I think pairing the unconstrained problems with constrained versions is a good experiment/eval design.

Critical weaknesses
  • Constraints proposed measure instruction following and tokenization, not creativity. Nothing here explicitly measures novelty.

  • Compliance is checked on outputs for some models, but on reasoning traces for others, so cross-model comparisons don’t make sense here.

  • The psychology framing (e.g. threats as “pressure,” dollars as “pleasure”) seems decorative, though I understand it was a motivator for the design.

Questions for authors

None

Reviewer_JYR4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors check LLMs' performance on creativity benchmarks under 1) Perturbation (e.g., renaming scene objects) 2) Constraint (e.g., length or banning a method). They discover that models degrade under these conditions. They additionally show that threat and reward sentences don't have a noticeable effect on performance of models in creative tasks, unlike in humans. In addition, the authors develop PCH, a harness that improves models' accuracy.

Critical strengths
  • The experiments are reported in detail (including prompts). LLM judges are compared against human labels.
  • A useful observation that models might try to reapply banned methods under different names
Critical weaknesses
  • The paper completely ignores the possibility of memorization, and that some models, like gpt-5.5 (whose cutoff is December 25) may have seen the presented data (e.g., Omni-Math which was made public on ~October 24... it's problems and solutions comes from math competitions, meaning that many were online even before that....). The degradation caused by perturbation further hints at that, so is the finding that models reach out to banned (memorized?) methods.
  • Iterative refinement and splitting ideation roles between agents are not a new concept. The paper should position PCH in relation to these lines of work (see https://arxiv.org/pdf/2305.14259 or https://arxiv.org/pdf/2410.09403 for instance).
  • In Table 2, for example, some models (like kimi) fail to comply on ALL problems under certain constraints. Even the baseline is not compliant for some models. This hints that the low performance might be an artifact of the setup. I suggest reporting correctness regardless of compliance in all your experiments as an additional metric.

In my opinion the paper would benefit from the following additions:

  • Elaborating on the datasets you use a bit more (Omega, MacGyver, CreativityBench). You do so in the related work, but I find that insufficient. Please also add an example or two of the original problems and their modification to the main text. That would greatly help with clarity.
  • Motivating the use of constraint theory as a useful novelty definition (e.g., show an example where an alternative solution is needed due to some limitation).
Questions for authors
  • How does solving problems under perturbation measure creativity?
Reviewer_B4umOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work examines LLM creativity by considering an operational definition of correctness in constraint-based problem solving. It introduces OMEGA-Constraint and Omni-MATH-Constraint, which couple mathematical problems with constraints on solution strategies, answer length, and characters. It explores sensitivity to solution-preserving perturbations, the impact of tool access and library restrictions, and reward/threat prompts. It proposes a new approach to improving performance using a proposer–composer harness (PCH) in which a typically smaller model generates a set of candidate responses, which a larger model then uses to compose its answer. Proposers are tested without additional training, with String Seed of Thought prompting, and with fine-tuning on a word-association task.

Evaluation benchmarks include 200-problem subsets and 60 DRAT items, supplemented with human validation, larger token budgets, proposer analysis, and baselines at inference time. Key findings include diminished joint correctness and compliance when imposing constraints, positive tool impact, limited reward/threat prompt impact, and improvements with PCH. The headline transfer results are obtained by choosing the best of ten proposers separately for each evaluation condition.

Critical strengths
  1. The paired eval reveals a meaningful downside

This pair analysis of constrained/unconstrained same problems is a nice approach especially for method constraints. They have alternatives methods defined in App G saying applicability is tested for each problem, and method judge validated with adjudicated human labels (87% overall agreement) in App H. Moreover, larger budget experiments help as well -- e.g. GPT-5.4-mini Omni-MATH has 57.0% baseline accuracy (33.0% for method restricted) for 16,384 tokens (all runs finish -- Table 3). This suggests hard to generate good solutions with method restrictions, other than due to small gen budget alone.

The robustness tests are nicely examined as well: App C shows repeat sampling and arit difficulty difference. Overall this is a decent empirical study for constrained problem solving.

  1. Small proposer assistants give credible improvements with evidence even in human eval.

Especially the DRAT result. Fine tuned 135M proposer pushes OLMo-2-32B judge up from 2.34 to 3.17 (Table 5). Nice that it goes in the right direction in the blind human audit: two annotators report 0.38/0.44 improvement for the audited sample (smaller than judge 0.77 improvement for that sample -- App E). Nice that you have random word/SelfRef comparisons in context.

Nice a math example beyond the two main composers. The reported Phi-4-mini-reasoning-to-GPT-4.1 setup pushes up OMEGA accuracy (all three constraints) from 9.0% to 25.0% ( Sect 4.4). Nice indication that generating extra (smaller model) draft is useful in practice, although mechanism and generality need some more clarification.

Critical weaknesses
  1. The evaluation does not demonstrate the alternative reasoning basis of the paper’s creativity perspective.

The essential criterion of the definition is to solve after the forbidden approach has been blocked. However, in Appendix H it is explicitly stated that a solution without displayed steps is judged to not have used the forbidden method, as it would have to be visible. Thus a solution is judged to use the allowed method even if it does not contain any evidence for an alternative derivation. Moreover, limited length and character counts effectively give authors degrees of freedom in the presentation, while still using the same mathematical method; in fact, Appendix D.2 explicitly says that variable renaming is allowed.

This problem is exacerbated by the fact that the evaluation does not address models that do not expose their reasoning in the same way; these only receive output verification, whereas exposed reasoning may be verified for additional models (§4.3). The ability to catch a forbidden solution therefore depends on the degree of reasoning that is exposed. The paper itself is aware of this problem, but that acknowledgment does not resolve the issue of comparability of the resulting creativity scores.

This problem also applies to the PCH result. Table 27 shows the composer benchmarked against an oracle of correct and compliant solutions. Surpassing this oracle shows something more than verbatim selection, but this could be achieved by condensing or reformatting an existing correct and non-compliant draft, without discovering a correct alternative mathematical approach that is permitted.

These points matter because the paper interprets the observed changes in joint correctness and compliance as changes in creativity, and contrasts this with human creativity. The experiments show interesting properties of constrained response generation, but there is insufficient evidence to back up the proposed comparison with human creativity and the claim of alternative reasoning.

  1. Maximal proposer selection and differing inference budgets weaken the evidence for the general PCH advantage.

Table 6 shows that the maximal proposer out of 10 alternatives is recorded for each condition. The authors correctly say this is an upper bound, and the means are given in Table 25, but the claim of better performance under each condition refers to this maximal selection. The means provide a more nuanced view, e.g. 50.0% for Qwen3-32B’s OMEGA method-only accuracy, and 49.4%, 47.8%, 47.5% for its plain, SSoT and tuned proposer-averaged accuracy. Appendix M gives significance tests after selecting the maximal proposer, relative to baseline, without comment on how this is accounted for. An independent proposer is required to demonstrate the reliability of the general transfer claim.

The sampling control is welcome, but the promised computational match is not delivered. Table 31 gives an OLMo sampling baseline with cost of 5.1 times a greedy pass, while PCH is 21.3 times, or more than 4 times the compute for PCH. Thus the PCH pass@20 outperformance does not demonstrate PCH advantage with equal inference budget. The reported saturation of self-consistency does not imply saturation of pass@k. The Qwen latency figure given in Appendix V is welcome, but it does not settle the question for the OLMo case.

These are particularly important as the idea of proposal followed by synthesis (and the multiple proposals from single proposers) is already present in Mixture-of-Agents and thus the value of PCH rests heavily on the value of the smaller-proposer setups and the task-constrained application. This related work is acknowledged in Appendix A, but not including the normal case on cost grounds doesn't make the case of being different to the same setup with the same models and budget.

Questions for authors
  1. need eval: evidence of allowed alternative method what proportion of successful method constrained responses have enough derivation to show that they used an alternative method? are the main results the same if you insist on having that derivation for success. also, if you compare similar visible results across models and insist on that derivation for success? what proportion of improvement do you get for pch compared to someone picking an answer out of candidates, with minimal transformations needed to meet output constraints, as a control?

  2. need clarification: character restriction feasible? in app D.2 you want the whole response to not use the banned character, but you also want \boxed{...}. one of the banned chars can be a variable (eg x), that is also needed in the required latex command. is it the case that character selection doesn't include characters required by the formatting or does the format compliance checker strip that formatting off? could you clarify what the actual rule is, and how many things are affected. right now it sounds unclear whether there might be conflicting requirements on some prompts.

  3. need eval: proposer is independent of test result how much of the improvements are there if you fix the proposer (and proposer mode) independently with different dev data. then measure it on the test problems reported. can the uncertainties include the selection of the proposer? table 25 is helpful, but it doesn't actually evaluate some procedure for selecting a proposer, prior to seeing the test results.

  4. need clarification and eval: inference budget matching how does the "compute matched" table 31 fit with the reported 5.1x sampling cost vs 21.3x pch cost (for olmo)? can you try to make this comparison again with the same stated resource budget? including actual proposer used, candidates produced, and composer prefill. this makes a huge difference for my opinion on the "it's more than just more inference compute for pch" claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.