- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper tests whether LLMs are able to produce responses that satisfy multiple constraints at the same time. The particular constraints are described only very generally in the paper but they appear to be things like: include a particular word, all words need to be unique, use markdown, include the word count, etc. The authors show that as the number of constraint increases, LLMs are unable to satisfy all of them at the same time, and increasingly make mistakes. For the stronger models, many of the failures are near misses (only one constraint was not satisfied).
Critical strengths
The dataset, while very narrow in scope (see below), is carefully constructed and the experiments are thorough. There are some interesting empirical tidbits (no interference between constraints; strong models can satisfy many constraints at the same time and only start to break down when more than 6 or 7 constraints need to be satisfied together).
Critical weaknesses
-
Abuse of page limit: oddly enough there is no table in the paper that lists the constraints. They are described quite vaguely in the main paper and then there's a link to two tables, neither of which appear to be actually in the paper (this probably means they're in the appendix, which I am not able to review because I have a lot of papers in my batch). Likewise, there is no discussion of limitations in the paper, but there is a link to the appendix. In my view, limitations are an important part of the discussion section of a paper, especially in a paper like this one that proposes a dataset that seems quite similar though perhaps somewhat better than previous datasets.
-
The particular operationalization of instruction following through formal constraints (counting words, writing acrostics, etc) is quite limited and has been included in a number of datasets already, so I wonder if there is substantial novelty or value in creating yet another dataset (albeit one that's harder than previous ones). I understand of course that more interesting or realistic constraints (in the sense that they might have more applications in the real world) pose verification challenges compared to easy-to-verify formal constraints, but perhaps our models are now strong enough that we can verify "loose" constraint satisfaction more effectively.
Questions for authors
Not a question but a minor grammatical comment - "interfere" cannot be used intransitively, so it should be "do constraints interference with each other", not "do constraints interfere".
Response to Reviewer T8Zo
We thank the reviewer for their careful reading and constructive comments, which have helped us clarify several points and improve the paper.
We agree that both belong in the main paper, and have added them. Section §3.1 now includes a table listing all 36 constraint types, grouped by dimension, with a one-line description of what each verifier checks. The full specifications remain in Table 7. We have also added a limitations paragraph at the end of §6, covering the points previously discussed in Section A: CSE examines one task family and one way of combining constraints, where all k requirements must be satisfied simultaneously; it includes only constraints that can be checked by deterministic verifiers; and results at k = 9 to 12 are based on 134 to 253 probes each.
We agree that formal constraints capture only a limited part of instruction following. We chose them because we need to verify responses reliably as the number of constraints increases. Since the paper measures how performance changes with k, we need to ensure that errors in verification do not also increase with k. An LLM judge would have to track all k constraints to assess a response, which is precisely the ability we are testing. FollowBench, for example, uses an LLM judge and evaluates up to five constraints (§1).
Even with deterministic verifiers, we found that careful auditing was necessary. An LLM review of near-miss failures identified four parsing issues, including one that incorrectly marked 1,051 responses as failures for R3 (§3.3). Extending the study to looser constraints using an LLM judge would be valuable, and our audited verdicts could provide a reference for calibrating such a judge. We have added this possibility to the limitations.
Our new analyses also show how these constraints interact in settings closer to practical prompting. For example, combining a structured output requirement, such as a JSON object, ranked list, or table, with restrictions on sentences or words can lower the pass rate of the latter by up to 28 pp (our reply to Reviewer evAu, Part 1).
Thank you for pointing this out. We have made the wording more explicit: "Do constraints interfere with each other?"