acceptodds
Under review as a conference paper at ICLR 2027

When Atomic Constraint Application Does Not Compose: Diagnosing Constraint Composition Failure in LLMs

Abstract

Recall does not establish that a model can apply a constraint. We define application-conditioned residual Constraint Composition Failure (CCF): failure to construct a feasible joint assignment after exact rule representation and isolated application have been demonstrated. Our gate exhaustively tests active operator inputs using the joint task's canonical context and complete-assignment interface. In 50-seed Matched Constraint Ring confirmations, residual failure at L2/L4 is .740/.935 for Qwen3.7-Max and .762/.867 for SenseNova-6.7-Flash-Lite. On seeds admitted at both lengths, Qwen increases by 21.7 points (paired 95% CI [6.5,37.0], p=.0129), whereas SenseNova's 7.9-point increase is inconclusive ([-7.9,23.7], p=.5078). A matched final-only repair instruction yields no reliable reduction. Repository-grounded evidence is model-family specific: DeepSeek-V4-Flash fails on 17/32 Updo and 4/24 Pebble control-success strict runs, whereas a jointly frozen Qwen extension succeeds on all 50 Updo runs but fails all 37 eligible Pebble runs. Deadlock and CodePlan audits delimit adjacent application, evaluation, and ordinary reliability failures. CCF is therefore a controlled diagnosis of an atomic-to-joint capability gap, not a universal depth law, model-wide deficit, software-task prevalence estimate, or renamed combinatorial explosion.

Primary area:
foundation or frontier models, including LLMs
Keywords:
constraint composition, large language models, capability-gated evaluation, compositional reasoning, code generation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_DhfNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper claims to investigate cases where an LLM can correctly apply individual constraints to the problem at hand, but fails to generate a solution that satisfies their composition/conjunction. The paper is so badly written that it is not clear that anyone could follow it, perhaps even the authors.

The target audience for this work is so narrow as to make this paper completely unnacceptable, it also means its hard to take away any general results from the work. This work appears to be the result of GenerativeAI which generates such a jargon filled mess that leaves it beyond human understanding.

Critical strengths

None

Critical weaknesses

The writing is in general incomprehensible.

The abstract is not standalone and essentially incomprehensible.

l.20. L2/L4 are meaningless here, and not ever really defined in the paper.

l.51. "enter the residual denominator" how are readers meant to understand this phrase.

l.53. "L2-to-L4 increase": meaningless

l.175. You define F(X,C) is you want to fix an X and C and use the notation F to represent F(X,C) you need to declare this.

l.176. The notation I(C) is not defined.

l.188. The "matched ring gate" is not defined here, and even later not properly defined.

Fig 1. What are "frozen strict denominator" and "stateless joint construction" they need discussion and explanation

l.210. Too terse to be comprehensible, e.g. "G_{pair} matched coordinator interfaces and non-joint checks"

l.218. Notation Y_{1:t} is not defined

l.233. The Matched Constraint Ring is not defined with enough detail to understand. Are we mean to understand that composing O_L ... O_2.O_1 gives the identity function, or that the composition has exactly one of the 5 symbols that for X_1 will map back to the same X_1 value.

l.239. "retains inventories" what is this meaning, we choose O_1, .., O_L so that there is exactly one fixpoint of the composition, i.e. one value of X_1 which is a fixpoint. Make this clear.

l.239. The notation L2 and L4 are not defined. Here we can guess you mean L = 2 and L = 4 in the definition above.

l.249. "One source is pinned" explain what this means.

l.256. What does it mean to freeze 50 paired seeds?

l.264. "Ring length is a dose variable" incomprehensible

l.274. "Updo" is never defined or given a reference

l.275. "Pebble" is never defined or given a reference

l.453 "T = 0" T is never defined or explained

Questions for authors

Can you define the research question you tackle clearly, and explain the results in terms of what we learn from them!

Post rebuttal:

The authors attempted to answer my questions clearly, this should have been done in the original submission. While the rebuttal does clear up some of the ambiguity, the underlying weakness of the experiment and lack of clarity still means this has to be rejected.

Reviewer_V6nBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Constraint Composition Failure (CCF) in LLMs, where a model can correctly represent and apply individual constraints but fails to generate a solution satisfying all constraints jointly. The authors propose a diagnostic framework that uses representation and atomic application gates to distinguish joint failures from individual constraint application failures.

The main experiments use a synthetic Constraint Ring benchmark, where LLMs must find a feasible assignment satisfying a set of cyclic constraints. The authors also extend the evaluation to two repository-grounded code coordination tasks and investigate whether a self-repair instruction can reduce CCF. The results show that joint failures can occur even after all atomic application tests are passed, while the effects of constraint length, code composition, and self-repair vary across models and tasks.

Critical strengths
  1. The paper provides a relatively rigorous diagnostic protocol that separates constraint representation, atomic application, and joint satisfaction. The use of deterministic verification and exhaustive testing of individual operator inputs helps reduce ambiguity in evaluating failures.

  2. The authors conduct additional code coordination experiments and boundary analyses to examine whether the observed failures extend beyond the synthetic Ring benchmark. The inclusion of negative results and unsuccessful controls helps clarify the limitations of their claims.

Critical weaknesses
  1. Limited novelty and scientific contribution. The main finding that successful atomic constraint application does not guarantee joint satisfaction is already well established in the compositional reasoning literature. The core Ring experiment is essentially a small-scale synthetic CSP, and the proposed diagnostic framework mainly provides a more elaborate way to demonstrate this known phenomenon. Although the authors introduce stricter gates and matched interfaces, it is unclear what new understanding of compositional reasoning these mechanisms provide beyond existing work. In particular, the experiments do not clearly distinguish constraint composition failure from ordinary multi-step reasoning or search failures.

  2. Insufficient experimental evidence and unclear generalization. The strongest results come from the synthetic Ring benchmark, whose task diversity is limited. The connection between Ring and the repository-grounded code experiments is mainly conceptual, without a formalized comparison or quantitative relationship. The code results also vary substantially across models and task families, without an analysis of which task properties explain these differences. Furthermore, the evidence for increasing CCF with Ring length is statistically significant for only one model, and the self-repair experiments examine only a limited intervention. Overall, the experiments do not sufficiently establish the generality or explanatory power of the proposed CCF framework.

Questions for authors
  1. Novelty relative to existing literature: Previous work on compositional reasoning has already established that single-step competence does not guarantee multi-step competence. What new scientific insight does the proposed CCF framework provide beyond this known observation and the capability-conditioned evaluation used in related work such as Composition Collapse?

  2. Interpretation of the Ring experiment: Since the Ring benchmark is a small synthetic CSP that can be solved by enumerating only five possible starting values, how can the authors distinguish the observed CCF from a failure to perform simple search or multi-step reasoning? What evidence supports interpreting these failures specifically as constraint composition failures?

  3. Generalization to code tasks: What is the formal or quantitative connection between the cyclic constraints in Ring and the state dependencies in the code coordination tasks? Given the substantial differences across models and task families, can the authors identify which task properties determine when CCF occurs? How do these results support conclusions beyond the specific tasks evaluated?

Reviewer_CAfMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether LLMs that can demonstrably apply each constraint in isolation can still satisfy the constraints jointly. The main testbed is a "Constraint Ring": L seeded permutations over 5 symbols arranged in a closed cycle with exactly one feasible assignment. An instance counts toward the residual only after two gates. The model must reproduce every lookup table exactly. It must also get all 5L single-step applications right, where each gate call uses the same context and the same full-assignment JSON format as the real joint call. Joint failures among the remaining instances are called residual CCF.

With 50 frozen seeds, residual CCF at L2/L4 is .740/.935 for Qwen3.7-Max and .762/.867 for SenseNova-6.7-Flash-Lite. The L2→L4 increase is significant only for Qwen. A one-call "check and revise privately" instruction doesn't help. Two code task families (Updo, Pebble) give a code analogue, conditioned on verified components and a successful non-interactive control coordinator. DeepSeek fails 17/32 and 4/24 there; Qwen fails 0/50 on Updo and 37/37 on Pebble. Several boundary audits (DeadlockGate, CodePlan, PatchGate, and two attempted non-interactive Ring controls) are reported mainly to show what is not CCF.

Critical strengths
  1. The matched application gate is a good idea. I think it is the real contribution. A standard objection to "models know X but fail at X+Y" results is that the atomic probe is easier for trivial reasons: shorter context, a different output format, a sampled subset of items. Here each atomic call gets the joint context and the joint output schema, all active source values are covered, and gate outputs never reach the joint call. Other evaluation work could reuse this.
  2. The reporting is also unusually honest. The Qwen Updo null is reported next to the Pebble result. Stopped controls are kept in, and the authors don't read anything into "significant for Qwen, not for SenseNova". I trust the numbers at the level they are claimed.
Critical weaknesses
  1. I don't think the residual is shown to be about composition. Solving a Ring instance means finding the one start X1 whose L-step propagation comes back to itself. In effect, it's a fixed-point search over 5 candidates. The gate certifies single forward lookups with the source pinned. It doesn't certify picking the start, chaining lookups in context, or checking closure. §3.2 admits all of these are inside the residual. Two of the paper's own results make me think they are most of it. The non-interactive comparator, the obvious control, came out harder than the interactive condition: 1/10 vs 9/10 success (Table 15). The crossover sits at 1/40, and the partitioned-cycle discovery is at the failure floor in both arms. So there is no evidence that interaction between constraints, as opposed to unscaffolded multi-step search with a JSON-only answer, is what drives failure. Then Table 7: whenever the model produces a correct five-start enumeration trace, joint failure is 0. Once the chaining is executed correctly, the answer is right. That looks like the familiar multi-hop reliability gap (Press et al. 2023; Dziri et al. 2023), not a separate layer. So the claim that survives is something like this: two hosted models, asked for JSON only, often can't output the fixed point of a 2- or 4-step permutation cycle over 5 symbols, despite getting every lookup right. At L4 they are below a random-start-then-propagate policy (.065/.133 vs .20). It's careful work, but I'm not convinced it's a new phenomenon. This is the main reason for my score.
  2. narrow evidence, and the code result looks like a spec-reading issue. The confirmation covers two hosted models (one a Flash-Lite), one synthetic task, and two lengths. I couldn't find whether thinking modes were on for Qwen/SenseNova or what the output limits were. Since Direct asks only for the final JSON, this matters a lot. On Pebble, all 49 strict Qwen interactive outputs fail the same assertion: the returned lanes don't all equal the terminal shared state. A perfectly uniform failure on one postcondition looks to me like the model consistently reading the return convention differently, not like a coordination breakdown. Much of the Pebble attrition is also models renaming sharedState (D.3). Unless it is shown that these coordinators get the state threading wrong, and don't just return it in a different shape, I can't count them as evidence for the same construct as Ring. Smaller point: RQ4 tests only one single-call repair prompt. The conclusion recommends search, explicit state, or solvers, yet none of these is run on the confirmation set.
Questions for authors

Q1. A.3 defines transition failures, non-closing trajectories and the collapse step, but they never show up in the results. Of the residual failures in the 50-seed confirmations, how many contain a transition the model got right in the gate, and how many are consistent propagations from the wrong start? This needs no new calls, and it matters most for my score. Mostly the latter would make it a search failure; mostly the former would be much stronger evidence. Q2. Thinking mode on or off for Qwen3.7-Max and SenseNova? What max_tokens? Do the logged token counts show hidden reasoning? For the 37 Qwen Pebble residuals: is the internal threading (durability → wait → callback) correct, with only the return convention wrong? Q3. What does re-scoring those two things separately give? Q4. Is there any equally specified non-interactive variant that models solve reliably? If not, how should a reader tell residual CCF apart from "an L-field JSON assignment under this prompt is just hard"?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.