- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies behavioral regression testing for stochastic LLM outputs. It applies repeated executions, itemwise one-sided Fisher exact tests, and Benjamini–Hochberg correction to identify tasks whose success probabilities may have decreased after a prompt or model update. The evaluation covers 24 settings across reasoning and code-generation benchmarks, supplemented by structured function-response experiments. The paper reports no observed discoveries in unchanged-version controls and substantial agreement with labels marking large observed success-rate drops. Comparisons with alternative decision rules, repetition-budget experiments, and a correction ablation characterize the procedure’s empirical behavior.
Critical strengths
The paper addresses a practical difficulty in LLM regression testing: a previously successful task can fail without any update. Repeated success counts and unchanged-version controls provide a useful framework for interpreting such failures, while multiple-testing correction explicitly addresses the risk of reporting spurious item-level regressions.
Critical weaknesses
-
The incremental practical value is insufficiently established. Although the underlying problem is relevant, the paper does not sufficiently demonstrate how the proposed procedure changes actual model-evaluation conclusions or deployment decisions relative to existing repeated-evaluation practices. Much of the motivation rests on familiar observations that LLM outputs fluctuate and that additional repetitions reduce sampling uncertainty. The paper would benefit from a concrete migration or update case showing how calibrated testing changes an actionable conclusion, beyond demonstrating improved behavior against selected heuristics.
-
Limited methodological novelty and depth of empirical insight. Fisher’s exact test and BH correction are established statistical methods, and the paper acknowledges closely related LLM migration work using similar ingredients. It introduces no substantial adaptation to the specific statistical challenges of LLM evaluation. Although it provides quantitative findings, many—including increased detection with larger samples and the trade-off introduced by multiplicity correction—are predictable from standard statistical principles. The contribution would be stronger with a clearer distinction from prior work or substantive new findings about LLM update behavior.
Questions for authors
Could the author clarify the motivation more concretely? That is, how exactly could this study help understand or change the established paradigm of model evaluation?