acceptodds
Withdrawn from ICLR 2027

Prompt Conditions Can Change Measured Intervention Effects in LLM Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Prompted language model evaluations often report the effect of an intervention under a single prompt condition. We test whether that reported effect changes when the surrounding prompt condition changes. On MMLU-Pro, a silent self-check increases exact-format accuracy for Gemma by under a standard prompt condition, under a masked condition, and under a token-matched metadata condition. A separate, pre-specified 300-task replication gives gains of , , and . The self-check gain is therefore larger under Standard than under Masked, with a simultaneous 95% confidence interval of . DeepSeek shows little observed difference across the same prompt conditions and Llama is inconclusive. The result also depends on scoring. On the replication set, an automated semantic answer scorer gives a Standard-minus-Masked difference of , while a separate constrained-decoding evaluation gives . Supporting experiments further show that average accuracy can remain almost unchanged even when many individual outcomes flip. These findings motivate reporting intervention gains under the prompt and scoring conditions that define them, rather than treating those choices as incidental details. Code: https://anonymous.4open.science/r/PCCMIE_ICLR27-BCA1/README.md

Primary area:
foundation or frontier models, including LLMs
Keywords:
language model evaluation, evaluation robustness, reference sensitivity, treatment effects

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_v9MKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies prompt‑by‑intervention interaction in LLM evaluation. It finds that different prompt conditions can greatly change the measured performance gain of the identical silent self‑check intervention, a phenomenon separate from standard prompt sensitivity. In pre‑specified MMLU‑Pro replication on Gemma‑12B, the self‑check gain differs by 20.00 pp between Standard and Masked prompts, while DeepSeek shows little such effect and Llama yields inconclusive results. This interaction remains positive under semantic scoring but is not reproduced in a separate constrained-decoding evaluation. Supporting experiments also show aggregate accuracy may stay stable despite widespread per‑example correctness flips. Overall, the paper recommends LLM intervention research to explicitly report intervention gains alongside corresponding prompt configurations, scoring rules and statistical uncertainty instead of treating prompt and scoring choices as trivial incidental details.

Critical strengths

This paper distinguishes prompt-by-intervention interaction from ordinary prompt sensitivity, showing that the measured effect of an intervention may depend on the surrounding prompt condition.

Critical weaknesses

(1) How does this work differ from prior studies of prompt sensitivity and evaluation variability? The novelty and contribution of this distinction seem unclear. The Masked and Metadata prompt conditions are artificially constructed contexts filled with semantically irrelevant content and may rarely be used in real-world LLM evaluations. Since these conditions replace meaningful auxiliary text with irrelevant or metadata-like content rather than using realistic semantic-preserving prompt variants, it is unclear whether the reported prompt-by-intervention interaction reflects practically meaningful sensitivity to plausible prompt variation or an artifact of an artificial prompt distribution shift.

(2) The empirical evaluation relies solely on one fixed silent self‑check intervention. No comparisons are made with alternative self‑check phrasings, chain‑of‑thought, explicit reflection, debiasing or other popular interventions. As a result, it is hard to tell whether the finding would extend beyond this intervention. In addition, the key interaction result is only validated on MMLU‑Pro. Only three models of roughly 8B‑12B scale are tested. A statistically‑significant interaction is found only in Gemma‑12B; DeepSeek shows little interaction and Llama‑3.1‑8B produces inconclusive results. It is unclear how model scale or architecture influences the observed phenomenon.

Questions for authors

Please see weaknesses.

Reviewer_AfXCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether the measured effect of an LLM intervention depends on the prompt condition used during evaluation.

Critical strengths

A useful evaluation question with reasonably careful statistical validation.

Critical weaknesses
  1. The main finding is substantially narrower than the broader evaluation problem motivating the paper.

  2. The main interaction is difficult to separate from answer-format compliance, which weakens its interpretation as an intervention effect on task performance.

Questions for authors
  1. Can the authors provide a more direct analysis of the prompt-by-intervention interaction using answer correctness independently of output-format compliance? In particular, how much of the 20 pp interaction remains when answers are correctly interpreted regardless of their formatting, and how reliable is the automated semantic scorer?

  2. The constrained-decoding experiment produces a qualitatively different result, including negative self-check gains across all prompt conditions. Can the authors explain why self-checking changes from strongly beneficial to harmful? Since both the task set and decoding procedure change, is there evidence that separates these two factors?

  3. How broadly do the authors expect the proposed evaluation concern to apply? The main interaction is replicated only for Gemma on MMLU-Pro, while the other two models do not provide comparable evidence. What supports extending the finding beyond this particular model, intervention, and prompt construction?

Reviewer_1HMVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper experimentally examines the actual effect of interventions on LLM prompts from the perspective that there exists an interaction with the auxiliary context contained in the base prompt, using a formulation that concretely discriminates the magnitude of this interaction. Taking the "silent self-check" intervention as a concrete subject of investigation, the paper analyzes it from various perspectives and reports all of the results. In the experimental results, in addition to showing that the magnitude of the intervention's effect differs across context types, the paper also analyzes differences across models and scoring methods.

Critical strengths
  • The definition of the interaction targeted by the experiments avoids ambiguity. The definition of Δ\Delta is identical with the definition of interaction in a standard two-factor experiment. It can be considered an easily interpretable quantity.
  • In addition to the concreteness of the experimental results, the paper follows the standard of experimental science: it discusses results obtained through a procedure whose experimental specifications were fixed in advance, and it discusses undesirable (negative/undesidable) results fairly. It also includes practical proposals such as Table 12. Therefore, I consider it to be content of high reference value.
Critical weaknesses
  • On the methodological side, the work can be regarded as a revisiting of methods already theoretically established in the field of experimental design, such as factorial designs, and its substantive contribution is limited to the empirical results. Specifically, the paper appears to deal with a two-factor full factorial design of 3 levels (context condition) ×\times 2 levels (intervention condition). However, even when considering multi-factor experiments that explicitly include additional surrounding factors such as model differences, there exist standard procedures for directly computing main effects and interactions numerically and performing statistical tests. From the perspective of experimental design, it seems unnatural that model differences are not included in the analysis framework and are only discussed qualitatively in the main text.

  • The dependence of prompt improvement methods on the base template has been widely discussed, and the related literature appears to be quite extensive. For example, [1] [2] [3] have already addressed similar topics. Given the aim of the paper, it would be preferable either to discuss existing studies within the framework of this paper and clarify their strengths and shortcomings, or to focus more on the novelty of the present work.

  • [1] Anton Voronov, Lena Wolf, Max Ryabinin. "Mind Your Format: Towards Consistent Evaluation of In-Context Learning Improvements." In Proc. ACL (findings), 2024.

  • [2] Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, Denny Zhou. "Large Language Models Cannot Self-Correct Reasoning Yet." In Proc. ICLR, 2024.

  • [3] Rem Hida, Masahiro Kaneko, Naoaki Okazaki. "Social Bias Evaluation for Large Language Models Requires Prompt Variations." In Proc. EMNLP (findings), 2025.

Questions for authors
  • The actual prompts appear to be included in the attached code, but I didn't review such content in-depth, and the validity of the content cannot be judged from the main text of the paper alone. Please include in the main text at least enough information for readers to understand the policy by which the prompts were written.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.