acceptodds
Withdrawn from ICLR 2027

What Can a Success-Set Flip Tell Us? Independent Response Validation in Language Model Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We ask which claims about items selected by a finite-budget success-set flip survive responses withheld from selection, larger discovery budgets, and later checkpoints. Among 151 Llama export-comparison flips selected at eight attempts, independent responses support 81 original directions; a separate pool supports 96, favors 21 opposite directions, and leaves 34 unresolved. Nine reversals survive Fisher/Holm correction over both directions of all selected items, with a median signed effect of percentage points. Across saved request-index prefixes, corrected determinate directions rise from 56 at depth 128 to 91 at 512. Increasing the discovery budget to 64 makes 120 selected items solved at both endpoints, including 57 with independent directional support. A separate Qwen3-4B checkpoint study follows 95 items with independently confirmed improvements. Their fixed-denominator score declines by points after , with a model-conditional posterior interval of , although 60 items retain support relative to baseline. An unmatched reference supplies descriptive context. Together, these fixed-item measurements connect a selected label to its later directional evidence under frozen machine scores. Training provenance and response dependence limit broader interpretation. The anonymous supplement provides analysis code and item-level results, which we will also release publicly.

Primary area:
learning theory
Keywords:
OPD、theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_C7nGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines what can be inferred when an item changes from unsolved to solved, or vice versa, within a limited response budget. It separates the responses used to select items from those used to evaluate their direction of change.

In the Llama comparison, nine of 151 selected items show an opposite direction that survives multiple-testing correction. Increasing the discovery budget also causes many items to become Retained despite independently supported differences in success probability. A separate Qwen checkpoint study follows 95 items with confirmed improvements and finds a subsequent decline of about 12 percentage points, while performance remains above baseline.

The paper distinguishes observed success-set transitions, changes in success probability, and persistence across checkpoints.

Critical strengths
  1. The separation of selection and validation responses is a meaningful strength. The paper tests whether fresh responses support the direction implied by an observed flip, rather than reusing the responses that selected the item. It distinguishes unresolved comparisons from evidence for the opposite direction and applies correction across both directions of all 151 selected Llama items. Table 2 provides the endpoint counts and effect sizes behind the strongest reversals. The budget analysis also gives a concrete illustration of how any-success labels can hide independently supported differences.

  2. The Qwen decline has substantial supporting evidence. A separate response pool shows a similar decline, and item-level changes agree closely across pools, with a Spearman correlation of 0.947. The decline becomes smaller but remains negative after removing the largest contributors, and it persists when any single domain is removed. The authors also acknowledge that the unmatched reference comparison is descriptive and does not establish a causal training effect. These checks strengthen the narrower conclusion about the selected items.

Critical weaknesses
  1. The contribution remains fairly narrow despite the extensive analysis.

The paper provides careful measurements of a familiar problem: finite-sample success labels can be misleading, and any-success coverage can conceal differences in success probability. The held-out direction checks add useful evidence, but it remains unclear how much broader understanding follows from these particular cohorts.

The strongest reversal findings concern one Llama comparison. The additional MATH-500 and D2 analyses show no opposite-direction support under the common rules in Table 24. Their different settings mean that this does not contradict the main result, but the paper offers limited guidance on when the problem becomes consequential in practice.

Similarly, the Qwen measurements establish a decline on the selected items, but do not explain whether the early gains reflect temporary peaks or a more persistent change in performance. The authors appropriately limit their claims, and the unmatched reference cannot resolve this question. I would like a clearer explanation of which practical conclusions would change because of this study, beyond the general recommendation to use held-out responses and report uncertainty.

  1. The primary reversal evidence is difficult to verify fully.

Appendix J states that the primary W evidence includes counts and response-membership records but not full answers. This allows numerical analysis of the counts but prevents direct inspection of what the scorer captured in the nine corrected reversals. Table 42 shows that 5 to 46 adverse score changes out of 1,024 would remove Holm support for individual reversals. This does not establish scoring errors, but it makes a targeted response audit valuable.

The unrecovered GAD training details and incomplete execution comparability also limit reproducibility. The dependence simulations identify a vulnerability of the statistical analysis, but the available partitions cannot determine the extent of dependence in the actual responses. The reported uncertainty is explicitly conditional on fixed items, fixed exports, and the response model; it should not be interpreted as evidence of reproducibility across training runs.

There are also presentation and reference issues that should be corrected. I repeatedly had to consult Table 7 to distinguish response pools from item cohorts. The switch between conditional Beta means and fixed-denominator yields makes closely related results harder to follow. For example, the two definitions produce 78.95% and 79.24% for C95 at t1. A main-text explanation of the pools, one worked example, and a consistent primary score would help. Figure labels are too small, and the gray subscript intervals in Table 2 are difficult to read.

Two reference entries contain mismatched information. Agarwal et al. (2024a) links to arXiv:2406.12009, which is FinTruthQA, while the intended distillation paper appears to be listed separately as 2024b. Toneva et al. (2019a) uses the title of Goodfellow and colleagues’ paper, “An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks” (arXiv:1312.6211), while the actual Toneva paper appears as 2019b. The authors should identify the intended sources, correct the entries and associated citations, and check the remaining bibliography. These errors do not by themselves invalidate the numerical results, but they raise concerns about the manuscript’s verification.

Overall, I lean toward weak rejection. The held-out measurements are careful and useful, but the limited additional insight and incomplete verification of the primary reversal evidence weaken the case for acceptance. Clearer practical implications and directly inspectable evidence for the decisive examples could change my assessment.

Questions for authors
  1. Can reviewers access the exact GAD export and the full primary W responses for the nine corrected reversals? Have these examples been checked manually to verify the scoring and clarify what behavior changed?

    Please also clarify whether the Native/GAD endpoints and the D, V, and W pools were evaluated under consistent inference settings. Were the models self-hosted or accessed through an external provider, and were precision or quantization, inference engine and backend, prompt templates, and decoding settings matched? Table 6 reports BF16 and some runtime details, but it is unclear whether these apply consistently across all three pools.

    If configurations differed, or the original responses cannot be recovered, could a targeted fresh evaluation under a common, documented setup reproduce the reversals while retaining the full responses and scorer outputs? This would help distinguish finite-sample label changes from differences introduced by the evaluation setup.

  2. D contains 120 responses per endpoint outside the original K = 8 discovery prefix. What directions do these responses support for the 151 selected items, particularly the nine W reversals? Reporting their counts, effect sizes, and uncertainty would help establish whether the contrary directions are also visible within the original response source.

  3. Could a selection-and-sampling model help explain the observed number and size of reversals using the actual discovery, validation, and multiple-testing procedure? A model fitted to W could provide descriptive context, provided the reuse of W and uncertainty in the estimated probabilities are acknowledged.

  4. What practical conclusion should readers change after this study? For the Qwen result, what can the available checkpoints establish about temporary early gains versus a more persistent decline? I recognize that this distinction is not required to establish the observed fixed-cohort decrease, but it matters for the broader lesson about improvement stability.

1 comment

Thank you for your feedback. In future work, we plan to develop a new method that builds on the findings of this paper and addresses the limitations identified in the evaluation. We will validate the method through additional experiments and compare it with relevant baselines.

Reviewer_CCabOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines whether finite-budget success-set flips reliably reflect changes in language model performance. The authors independently validate items labeled Learned or Forgotten in a Llama model comparison and track previously confirmed improvements across existing Qwen checkpoints. They find that some labels contradict held-out directional evidence, larger sampling budgets can obscure differences in success probabilities, and some confirmed gains decline at later checkpoints. The appendix includes controls, statistical simulations, and an additional training experiment that did not meet its prespecified diagnostic criterion.

Critical strengths
  1. The problem is practically relevant and supported by concrete counterexamples. The paper identifies items whose independently measured changes oppose their original labels, including cases that survive multiple-testing correction. These results illustrate the risks of interpreting success-set flips directly as learning or forgetting.

  2. The paper includes useful controls and reports negative findings. Separating selection from validation, conducting same-model controls, and analyzing unresolved scores strengthen parts of the empirical evidence. The authors also report settings without supported reversals and acknowledge that the training diagnostic did not meet its prespecified criterion.

Critical weaknesses
  1. The novelty and incremental empirical contribution remain limited. The underlying statistical ideas are established, and the paper lacks systematic, equal-budget comparisons on real tasks that demonstrate the additional value of its validation procedure.

  2. The experimental narrative is fragmented. Different models address different questions, and the connection between the main experiments, appendix diagnostics, and claimed contributions is insufficiently clear. Training runs, repeated evaluations, and simulations also need to be distinguished more prominently in the main presentation.

Questions for authors
  1. Compared with existing Bayesian evaluation or directly analyzing the full success counts, what additional benefit does the proposed procedure provide under the same total sampling budget?

  2. Why are Llama and Qwen assigned different main experiments? Could a common protocol be applied across multiple models to test whether the same conclusions hold?

  3. Given that FitFunc did not meet its prespecified diagnostic criterion, which central claim does this experiment support?

1 comment

Thank you for your feedback. In future work, we plan to develop a new method that builds on the findings of this paper and addresses the limitations identified in the evaluation. We will validate the method through additional experiments and compare it with relevant baselines.

Reviewer_q9QdOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper looks at success set labels and examines how reliable observed changes across different model checkpoints are at actually reflecting changes in a language model's probability of solving an item. The work performs an empirical study of Llama and Qwen checkpoints and found large changes to conclusions about items' learned/forgotten/retained status when new responses were sampled or the response block was expanded.

Critical strengths

The paper shows that a common evaluation protocol is noisy and sensitive to design choices (namely, discovery budget size)

Critical weaknesses
  • It's a nice finding but the scope of the paper and of the experiments is very limited. There isn't really a contribution aside from pointing out that a protocol isn't robust
  • The presentation in section 2 feels unnecessarily complex for a simple problem setting. While its nice to formalize the statistical assumptions/protocol, several tests are being used without the underlying assumptions of those tests being justified (e.g., the reader needs to trust without explanation that the Fisher/Holm correction is applicable and the right approach)
  • Empirical results are narrow, covering only two models in a rather specific setting
  • The abstract in particular was difficult to read. It is currently just a stream of numbers, using variables that aren't actually defined. Those numbers are rather specific to the cases studied here and there really aren't higher level lessons or takeaways
  • I generally found the results difficult to follow. There often wasn't a clear distinction between primary and sensitivity analyses, what the main estimand in each experiment was, or just a simple concrete explanation of what each statistical result menat
Questions for authors

Do you have recommendations for evaluation protocol aside from just reporting the settings? E.g., how a finite amount of compute should be allocated (discovery budget vs. evaluating on more questions) to best ensure robustness? Something like that would perhaps be a more concrete takeaway for the reader How did you rule out that the reversals you're observing aren't just coming from different settings/errors in your generation or scoring procedures compared to those of the original response set?

1 comment

Thank you for your feedback. In future work, we plan to develop a new method that builds on the findings of this paper and addresses the limitations identified in the evaluation. We will validate the method through additional experiments and compare it with relevant baselines.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.