- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies whether changes in answers across prompts help predict a model’s loss of accuracy on different data. The authors test fifteen language models on three classification tasks and seven dataset shifts. They measure changes in accuracy and disagreement between answers across ten wordings. They test whether adding these measures improves predictions for model families excluded from training. The measures help in two test groups and hurt in another, leaving their usefulness for new model families unclear.
Critical strengths
-
The paper asks a useful practical question. Predicting how much accuracy a new model will lose could reduce the need to test every model on every dataset.
-
The study covers several models and tasks. It evaluates fifteen models, three classification tasks, and seven dataset shifts.
Critical weaknesses
-
The writing makes the main findings harder to assess. Long sentences and repeated labels interrupt the explanation. The reader must work through many qualifications to understand what each experiment establishes and why it matters.
-
The paper adds two simple measures to a standard regression model. Its value therefore rests mainly on the empirical findings, which do not yet give a clear explanation or a reliable prediction rule.
-
The paper gives limited guidance on when sensitivity will help. It reports improvements in two groups and worse predictions in another. The checks identify factors that change the results, but they do not give a clear way to predict which outcome to expect for a new family.
-
The predictor is trained on only five model families. Removing the Qwen 3 family changes both sensitivity measures from hurting to helping in the first held-out group. This makes that result difficult to treat as stable.
-
Different kinds of prompt changes are combined into one measure. The wordings change label order, whitespace, wording, and other features, sometimes together. The paper does not isolate which changes provide useful information. This limits what we learn about the source of the measured sensitivity.
Questions for authors
-
Does either sensitivity measure improve predictions when the baseline already includes both source accuracy and the invalid response rate? This would help establish what sensitivity adds beyond response formatting.
-
How much do the results change when the evaluation items are resampled? The current checks vary the prompt wordings, but do not show how much the findings depend on the particular 180 items selected.
-
How does adding prompt sensitivity compare with adding model size to the baseline, using the same held-out groups? This would help establish its value over a simpler model property.
-
Under the same number of model calls, is measuring sensitivity more useful than evaluating more source examples to estimate accuracy? This comparison would help establish whether repeated prompting is a useful use of the evaluation budget.