- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper asks what instruction-compliance scores for LLM verifiers actually measure when a rule conflicts with the factually correct answer. It notes that the usual conflict-only score mixes sensitivity to the rule with sensitivity to correctness. It then collects all four combinations of compliance and correctness by flipping which option the rule designates. Experiments use four instruction-tuned models between 3B and 8B parameters on two-option world-knowledge questions. Rules that quote the permitted answer can give quite different scores from rules that refer to it by a label. When the label and the quoted string disagree, the models tend to follow the quoted string, even in a variant with invented words and no factual question.
Critical strengths
-
The methodology is unusually transparent. Hypotheses and thresholds are registered in advance, contrasts are made within items, and negative results are reported rather than dropped.
-
The pseudoword experiment shows that, for these models, a quoted string wins over a label binding they can otherwise resolve. The reminder to check whether a model can parse a rule at all, before reading its compliance score, is also useful for evaluation design.
Critical weaknesses
-
The significance is limited. The two-option setup with labelled options is not used by any of the benchmarks the authors themselves surveyed. The property-rule arm, which is the closest to real benchmarks, is inconclusive because the models are near chance on it. The paper does not show that any published benchmark or LLM judge is affected. Under the quoting and label rules the aligned cells are near ceiling, so there the four-cell design adds little beyond the conflict score. Auditing all four cells also gives a null result in the authors' own selection experiment. Since the central identity is standard signal-detection algebra, as the authors acknowledge, the contribution rests entirely on these narrow experiments.
-
The paper is very hard to read. It uses many custom names, refers to an unpublished earlier study, and spends much of its space on registration bookkeeping, while the verifier wordings are not given. The formatting also has clear problems. The paper does not use the ICLR template's font and heading style, and Figures 2–4, which the main text relies on, are placed deep in the appendix. About ten appendix pages are raw Markdown, with one table running off the page, and several references list only "et al." as authors.
Questions for authors
- Do the effects hold for larger models that are commonly used as judges?
- Can you report results based on the probability of answering yes, with a bias-aware analysis?
- What are the exact verifier wordings used in the experiments?
As the authors did not respond during the discussion period, I will keep my original score.