- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 2(+1)
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces an analysis over multi-constraint instruction-following tasks, specifically suggesting a finer-grained metric that tracks how different constraints are satisfied across different levels of prompt complexity. The suggested metric differentiates between different types of LLM errors over constraints - for example, a constraint that was successful with a simpler prompt (level k) but fails when a new constraint is added to the prompt (level k+1), versus a mistake on the newly added constraint. They verify the performance both via an LLM as a judge, human annotations, and rule-based evaluations. They evaluate a few different LLMs such as Llama and Qwen.
Critical strengths
- The setup is highly relevant to LLM evaluation, as it reflects real use cases of users.
- The finer grained metric is novel and can be useful to analyze LLM behavior.
Critical weaknesses
-
The paper does not back up many of its claims. There is a severe lack of empirical backing, with a lot of vague sentences, missing examples, and missing citations to back up statements that the authors make.
-
The paper mentions scores which are not clear. For example, the abstract mentions metrics of "-1.44 and +1.20" without clarifying what these numbers mean, what the base score is, or how they are calculated. Forcing the reader to guess the units or the mathematical basis of the primary findings in the abstract is a major writing issue.
-
The terminology is confusing and not standard (e.g., level?). Vague terms are introduced with little clarity, obscuring the basic testing environment on the first few pages. Furthermore, the transition-based vocabulary can mislead a reader into thinking the evaluation is a multi-turn conversation, when in reality each level is a completely independent, single-turn prompt.
-
Human evaluation is mentioned, but the setup in which humans annotate is not explained. The paper relies on human annotations to audit constraint extraction and judge errors, but it fails to describe the human annotation pipeline. There are no details on who the annotators were, what instructions or guidelines they were given, how many annotators graded each item, or how annotator disagreement was resolved. This makes the human validation impossible to replicate or verify.
-
The writing of the paper makes it really difficult to follow. At certain points it sounds more like prose than scientific writing. A lot of the sentences remain vague and not explained properly, and further context is missing.
Questions for authors
-
Is the setup that the model was tested on single turn or multi turn? what is the meaning of "level" in your setup?
-
Could authors provide concrete examples of the setups in which you evaluate models? grounding the terminology used in the paper (e.g., "level") to actual prompts, and showing how "broken" or "acquired" will look like in the data
3 comments
I thank the authors for the detailed response. The revised version is much clearer now. However, some of my concerns about soundness remain. The FollowBench null result seems fragile (flipping a few labels changes it) and relies on a judge with quite high error rates, and the ManyIFEval effects are small. Overall, I’m raising my score to reflect the improvements.
Thank you for revisiting the revised manuscript and for reconsidering your assessment. We appreciate your recognition of the improved clarity.
We agree that the FollowBench estimates are instrument-dependent, which is why we now limit their interpretation explicitly and report the adversarial-flip sensitivity. One clarification: the small flip counts come from an adversarial oracle that selects, with knowledge of the fitted trend, the changes that move it most. Under the non-adversarial perturbations we tested, random flipping at rates up to 50% and level-dependent corruption rising from 5% to 35% produced no positive trend intervals. Base-rate randomization at 50% produced rare positive intervals (fewer than 5% of draws for two models). These simulations are not calibrated to the actual judge-error process, so we do not interpret them as ruling out other error patterns (Appendix T).
We also agree that the ManyIFEval effects are modest in magnitude. Our main conclusion is not that adding constraints universally causes large increases in breakage, but that marginal satisfaction and conditional breakage answer distinct questions even under deterministic scoring.
We appreciate your careful engagement and have retained these limitations explicitly in the revised paper.
Thank you. Your two questions identified a real failure in the submission: the setup was not stated clearly enough for a reader to reconstruct it. We answer both here and have revised the paper.
Q1: Is the setup single-turn or multi-turn?
Single-turn, with independent generations. Each level is a separate prompt. The model is shown level k on its own, with no history of earlier levels and no dialogue, and generates a fresh response. A transition compares judgments about the same requirement in two independently generated responses. It is not a conversational turn, a memory test, or a model revising its own earlier answer.
You were right that the submission invited the opposite reading. The phrase "single-turn" appeared once in the whole paper, in the Limitations. It is now stated in Section 2 (p. 2), where the setup is introduced.
Q2: What is a level, and what do breakage and acquisition look like?
A trajectory is a sequence of prompts designed to add one requirement per level, though surrounding wording can also change. New Figure 1 (p. 3) shows a real FollowBench trajectory rather than notation:
For Llama-3.1-8B the judge records the Mr. Rogers constraint as satisfied at level 1 and not at level 2, which is breakage, while the new constraint is satisfied, which is acquisition. Panel (b) shows the four outcomes for a previously satisfied constraint and states that recovery is not in that table, because it conditions on a previous failure.
Appendix Z (p. 25) gives the same example with excerpts from both actual responses, so the verdict can be inspected: level 1 opens "Neighbor, I'm so glad you asked", level 2 opens "Dear friend, I see that thou dost struggle with the weight of stress". We say there that Style has no rule-verifiable coverage, so this is the judge's verdict and is not independently certified, and that a reader may reasonably think some of the original register survives. This illustrates why we distinguish recorded verdicts from independently verified outcomes. Section 4.4 quantifies judge disagreement on a separate rule-verifiable subset, which contains no Style constraints.
On the human annotation setup
Section 4.3 (p. 5) now gives the protocol in the main text. Two annotators labelled the 392 constraints entering the breakage pool as correct, incorrect or ambiguous, under a plan frozen before labelling. One is an author of this paper and labelled all 392; the second is not an author, labelled 155 for agreement, and was compensated. The interface showed neither model outputs nor the other annotator's labels, but both could view a word-level diff of the adjacent prompts, which is the signal the extractor itself uses, so the check is not fully independent of extraction. Ambiguous labels and unresolved disagreements count as not confirmed; agreement on the 155 shared items is 77.4% with Cohen's kappa 0.394, and no adjudication was performed.
This study validates constraint extraction, not the correctness of the judge's response-level verdicts. Those are assessed separately against deterministic verifiers in Section 4.4.
On the abstract's numbers
Corrected. The abstract now says the slopes are percentage-point changes in the probability that the tracked instruction is satisfied, per added instruction, and gives the base for the one model that declines detectably: Qwen, 92.6% satisfied alone and 88.0% with four more added.
On empirical backing
The supplement contains both annotators' raw labels, the frozen protocols with their dated amendments, generation provenance and configuration audits, and a numerical claim ledger that recomputes 108 reported values from the frozen artifacts and fails if any disagrees. We would be glad to clarify the evidence or provenance behind any particular result that remains unclear.