acceptodds
Under review as a conference paper at ICLR 2027

A quoted answer overrides the rule that designates it: what an instruction-compliance score measures depends on how the rule names its answer

Abstract

Conflict-only instruction-compliance scores ask whether a verifier endorses the rule-designated answer when the facts favour another. Their contrast, , subtracts correctness sensitivity from compliance sensitivity; without aligned items, it cannot identify the two separately. On four fixed instruction-tuned checkpoints, holding label indirection fixed and printing the permitted answer changes conflict-cell error by , with substantial variation across checkpoints. We then separate the rule's label-based designation from the answer it quotes. In a prospectively registered form where the two disagree, and with the factual question removed and both options replaced by invented words, accuracy against the label binding is 0.103, versus 0.955 when they agree. The quoted option is endorsed at 0.806, versus 0.012 for the label-designated one. The label binding remains usable without a quotation (balanced accuracy 0.922); thus factual-answer recall cannot explain this override of the label-based designation in these constructed rules. For property-based rules, balanced accuracy without a factual question is 0.526 [0.518, 0.535]: below the registered competence floor, but not at chance under its registered criterion. The verdict is indeterminate, so their conflict-cell cost cannot be isolated as a rule-form effect. An independent equal-budget comparison also did not establish a selection advantage for the four-cell audit. These empirical findings concern this fixed small-model roster and two-option tasks; the identification problem applies to the score's definition.

Primary area:
foundation or frontier models, including LLMs
Keywords:
instruction compliance, LLM evaluation, verifier models, measurement validity, instruction following

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_SC8BOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper asks what instruction-compliance scores for LLM verifiers actually measure when a rule conflicts with the factually correct answer. It notes that the usual conflict-only score mixes sensitivity to the rule with sensitivity to correctness. It then collects all four combinations of compliance and correctness by flipping which option the rule designates. Experiments use four instruction-tuned models between 3B and 8B parameters on two-option world-knowledge questions. Rules that quote the permitted answer can give quite different scores from rules that refer to it by a label. When the label and the quoted string disagree, the models tend to follow the quoted string, even in a variant with invented words and no factual question.

Critical strengths
  1. The methodology is unusually transparent. Hypotheses and thresholds are registered in advance, contrasts are made within items, and negative results are reported rather than dropped.

  2. The pseudoword experiment shows that, for these models, a quoted string wins over a label binding they can otherwise resolve. The reminder to check whether a model can parse a rule at all, before reading its compliance score, is also useful for evaluation design.

Critical weaknesses
  1. The significance is limited. The two-option setup with labelled options is not used by any of the benchmarks the authors themselves surveyed. The property-rule arm, which is the closest to real benchmarks, is inconclusive because the models are near chance on it. The paper does not show that any published benchmark or LLM judge is affected. Under the quoting and label rules the aligned cells are near ceiling, so there the four-cell design adds little beyond the conflict score. Auditing all four cells also gives a null result in the authors' own selection experiment. Since the central identity is standard signal-detection algebra, as the authors acknowledge, the contribution rests entirely on these narrow experiments.

  2. The paper is very hard to read. It uses many custom names, refers to an unpublished earlier study, and spends much of its space on registration bookkeeping, while the verifier wordings are not given. The formatting also has clear problems. The paper does not use the ICLR template's font and heading style, and Figures 2–4, which the main text relies on, are placed deep in the appendix. About ten appendix pages are raw Markdown, with one table running off the page, and several references list only "et al." as authors.

Questions for authors
  1. Do the effects hold for larger models that are commonly used as judges?
  2. Can you report results based on the probability of answering yes, with a bias-aware analysis?
  3. What are the exact verifier wordings used in the experiments?
Reviewer_HtUyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether instruction-compliance scores are affected by how rules are worded. The authors use a four-cell evaluation design and vary rule forms, including quoted answers and label-based designations, to distinguish models’ preferences for rules, factual answers, and quotations. Experiments on four small models show that rule wording substantially affects evaluation results; even when factual questions are removed and options are replaced with pseudowords, models still tend to endorse the quoted answer.

Critical strengths
  1. The research question is valuable. The paper shows that high instruction-compliance scores do not necessarily indicate that models understand and apply the rules, offering useful insights for evaluation design.

  2. The controlled experiments are fairly comprehensive. Label-based rules, quotations of forbidden answers, and pseudoword options help rule out several alternative explanations.

  3. The experimental materials are complete and the results are reproducible. The authors provide raw responses and analysis code, and transparently report negative results and limitations.

Critical weaknesses
  1. The interpretation of the central experiment remains ambiguous. When the label and quotation designate different answers, the rule itself is contradictory. Choosing the quoted answer does not necessarily indicate a failure to follow the rule; the model may interpret the quotation as correcting the label binding.

  2. The practical impact remains unverified. The experiments cover only four 3B–8B models and synthetic two-option tasks. I suggest including a stronger model and validating the phenomenon on a real-world model-judge evaluation task.

  3. The paper’s organization is somewhat complex. There are many rule abbreviations and metrics, and the central results are scattered across the main text and appendices. A table comparing the key rule forms would help, and the main pseudoword results should be included in the main text.

Questions for authors
  1. Do models still prefer the quoted answer when the order of the two contradictory rules is reversed?

  2. Does the central finding hold when the instructions explicitly state that the label binding takes priority over the quotation?

  3. Does this phenomenon affect scores or model rankings in real-world evaluation tasks?

Reviewer_sJKwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates whether high performance on instruction following means that the model actually follows and understands the instructions or simply follows the rules. The authors suggest that we should look at two axes: whether the answer is factually correct or wrong, and whether it follows the instruction.

Critical strengths
  • The paper conducts various analyses to show that performance depends not only on how the model understands the rules but also on how the answer is represented.
Critical weaknesses
  • Experiments are conducted in a synthetic setup with two choices. It is not clear how meaningful these choices are. Did you check whether the models are doing random guess or is doing meaningful choice between the two?
  • I’m not sure if we can say that quoting the answer is a shortcut in Section 4.2. Are you suggesting that this is incorrect behavior?
  • The writing has several serious issues. For example, what are p10p_{10} and RCR_C in the abstract? Overall, the paper seems like a list of observations without sufficient summarization or organization.
Questions for authors
  • How meaningful is the (2 x 2) design suggested by the authors? What insights would these results provide to the community?
1 comment

As the authors did not respond during the discussion period, I will keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.