- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies prompt‑by‑intervention interaction in LLM evaluation. It finds that different prompt conditions can greatly change the measured performance gain of the identical silent self‑check intervention, a phenomenon separate from standard prompt sensitivity. In pre‑specified MMLU‑Pro replication on Gemma‑12B, the self‑check gain differs by 20.00 pp between Standard and Masked prompts, while DeepSeek shows little such effect and Llama yields inconclusive results. This interaction remains positive under semantic scoring but is not reproduced in a separate constrained-decoding evaluation. Supporting experiments also show aggregate accuracy may stay stable despite widespread per‑example correctness flips. Overall, the paper recommends LLM intervention research to explicitly report intervention gains alongside corresponding prompt configurations, scoring rules and statistical uncertainty instead of treating prompt and scoring choices as trivial incidental details.
Critical strengths
This paper distinguishes prompt-by-intervention interaction from ordinary prompt sensitivity, showing that the measured effect of an intervention may depend on the surrounding prompt condition.
Critical weaknesses
(1) How does this work differ from prior studies of prompt sensitivity and evaluation variability? The novelty and contribution of this distinction seem unclear. The Masked and Metadata prompt conditions are artificially constructed contexts filled with semantically irrelevant content and may rarely be used in real-world LLM evaluations. Since these conditions replace meaningful auxiliary text with irrelevant or metadata-like content rather than using realistic semantic-preserving prompt variants, it is unclear whether the reported prompt-by-intervention interaction reflects practically meaningful sensitivity to plausible prompt variation or an artifact of an artificial prompt distribution shift.
(2) The empirical evaluation relies solely on one fixed silent self‑check intervention. No comparisons are made with alternative self‑check phrasings, chain‑of‑thought, explicit reflection, debiasing or other popular interventions. As a result, it is hard to tell whether the finding would extend beyond this intervention. In addition, the key interaction result is only validated on MMLU‑Pro. Only three models of roughly 8B‑12B scale are tested. A statistically‑significant interaction is found only in Gemma‑12B; DeepSeek shows little interaction and Llama‑3.1‑8B produces inconclusive results. It is unclear how model scale or architecture influences the observed phenomenon.
Questions for authors
Please see weaknesses.