acceptodds
Withdrawn from ICLR 2027

Is It a Regression, or Just Noise? Calibrated Behavioral Regression Testing for Stochastic LLM Behavior

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An agent can fail a previously successful task even when its model and prompt are unchanged. Regression testing must separate this rerun variation from evidence that an update reduced reliability. We examine a repeated count test that compares each item's success counts with a one sided test and corrects across the suite for multiple comparisons. Our evaluation distinguishes two questions: unchanged version splits measure spurious alarms, while update runs measure agreement with labels marking large observed success rate drops. The test raised no observed alarms in unchanged version controls and selected many of the positive labels after updates. A structured function response study, repetition budget sweep and correction ablation show how the results depend on workload and sampling. Because the update labels share the tested runs, their agreement characterizes an empirical screen; independent reference runs remain necessary to measure detection of latent probability decreases.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, regression testing, stochastic evaluation, Fisher test, multiple testing, software reliability

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_rJN2OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies behavioral regression testing for stochastic LLM outputs. It applies repeated executions, itemwise one-sided Fisher exact tests, and Benjamini–Hochberg correction to identify tasks whose success probabilities may have decreased after a prompt or model update. The evaluation covers 24 settings across reasoning and code-generation benchmarks, supplemented by structured function-response experiments. The paper reports no observed discoveries in unchanged-version controls and substantial agreement with labels marking large observed success-rate drops. Comparisons with alternative decision rules, repetition-budget experiments, and a correction ablation characterize the procedure’s empirical behavior.

Critical strengths

The paper addresses a practical difficulty in LLM regression testing: a previously successful task can fail without any update. Repeated success counts and unchanged-version controls provide a useful framework for interpreting such failures, while multiple-testing correction explicitly addresses the risk of reporting spurious item-level regressions.

Critical weaknesses
  1. The incremental practical value is insufficiently established. Although the underlying problem is relevant, the paper does not sufficiently demonstrate how the proposed procedure changes actual model-evaluation conclusions or deployment decisions relative to existing repeated-evaluation practices. Much of the motivation rests on familiar observations that LLM outputs fluctuate and that additional repetitions reduce sampling uncertainty. The paper would benefit from a concrete migration or update case showing how calibrated testing changes an actionable conclusion, beyond demonstrating improved behavior against selected heuristics.

  2. Limited methodological novelty and depth of empirical insight. Fisher’s exact test and BH correction are established statistical methods, and the paper acknowledges closely related LLM migration work using similar ingredients. It introduces no substantial adaptation to the specific statistical challenges of LLM evaluation. Although it provides quantitative findings, many—including increased detection with larger samples and the trade-off introduced by multiplicity correction—are predictable from standard statistical principles. The contribution would be stronger with a clearer distinction from prior work or substantive new findings about LLM update behavior.

Questions for authors

Could the author clarify the motivation more concretely? That is, how exactly could this study help understand or change the established paradigm of model evaluation?

Reviewer_dYbjOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates behavioral regression testing for stochastic large language models, where previously successful tasks may fail upon repeated execution even without model or prompt updates. The authors study a statistical testing framework that compares per-item success counts across two model versions using a one-sided Fisher's exact test, followed by Benjamini–Hochberg (BH) correction to control the false discovery rate across an evaluation suite.

Critical strengths
  1. This paper addresses an important problem in LLM reliability evaluation by distinguishing genuine behavioral regressions from stochastic execution noise. This problem is highly relevant to model updates, prompt modifications, and the reliable deployment of LLM systems.

  2. Comprehensive experimental evaluation. The experiments cover multiple datasets, models, and update scenarios. The authors also investigate unchanged-version controls, repetition budgets, and BH correction through ablation studies, providing useful empirical insights into stochastic LLM regression testing.

Critical weaknesses
  1. The proposed method primarily combines Fisher's exact test and BH correction, both of which are well-established statistical techniques. The paper does not demonstrate a sufficiently clear methodological breakthrough or provide strong novel empirical findings beyond existing approaches.

  2. The same execution outcomes are used to construct regression labels and perform statistical testing, introducing dependence between the evaluation metric and the detection method. Therefore, the reported 85.7% agreement does not sufficiently demonstrate the ability to identify genuine behavioral regressions.

  3. The unchanged-version controls use 10+10 executions, while the update comparisons use 20+20 executions. Since statistical testing depends on sample size, the observed zero false-positive rate does not directly establish reliable false-positive control under the actual update evaluation setting.

  4. The negative-flip baseline employs different decision rules for unchanged-version controls and update comparisons. Additionally, the matched agreement is obtained through retrospective threshold selection, potentially introducing selection bias. These issues weaken the evidence supporting the proposed method's superiority over the baselines.

Questions for authors
  1. The same execution outcomes are used to construct regression labels and perform statistical testing. Could the authors use independent evaluation samples to establish reference labels and further validate whether the proposed method can detect genuine behavioral regressions rather than merely identify observed success-rate drops?

  2. The unchanged-version controls and update comparisons use different execution budgets, while the negative-flip baseline employs different decision rules across these settings. Could the authors reevaluate the methods under consistent sampling budgets and decision criteria to verify whether the proposed method's advantages still hold?

Reviewer_9kDuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies behavioral regression testing for stochastic LLMs. It proposes comparing repeated binary outcomes after an update using item-wise Fisher tests, followed by Benjamini–Hochberg correction across the evaluation suite. The authors evaluate the approach across benchmark datasets, models, and update types, and examine the effects of repetition budget and multiple-testing correction.

Critical strengths

The paper addresses a relevant practical problem.Single failed run after an update may reflect stochastic variation rather than a genuine regression.

The experiments cover multiple datasets and model families. The discussion of how the repetition budget affects the resolution of the statistical test is useful.

Critical weaknesses

The novelty and contribution are insufficient. The proposed approach combines established statistical tool.Fisher’s exact test and Benjamini–Hochberg correction—and the paper acknowledges prior work that already uses a similar combination for item-level LLM regression analysis. The additional experiments do not establish a significantly new insight beyond this prior work. Overall, it is unclear what methodological or empirical contribution rises to the level expected at top conference like ICLR.

The comparisons and reported results are difficult to interpret. The matched summary selects thresholds retrospectively using both the control results and the update labels. It also assigns zero agreement to baselines that do not satisfy the control-rate cap, which substantially affects the aggregate comparison. The negative-flip baseline is summarized differently in the unchanged-version and update experiments, making its results difficult to compare directly. In addition, the repetition-budget experiment redefines the positive labels at each budget, so it does not show how power changes against a fixed set of target regressions.

The presentation makes the empirical claims difficult to assess. The limitations of the central metrics and comparisons are scattered across the paper rather than made clear where the results are introduced. In particular, the paper emphasizes agreement with labels derived from the same runs used by the test, even though this agreement cannot be interpreted as independently measured regression-detection performance. The tables and figures also do not clearly convey what each comparison establishes. As a result, the reader must piece together substantial qualifications to understand the reported findings.

Questions for authors

see the weakness above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.