- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines what can be inferred when an item changes from unsolved to solved, or vice versa, within a limited response budget. It separates the responses used to select items from those used to evaluate their direction of change.
In the Llama comparison, nine of 151 selected items show an opposite direction that survives multiple-testing correction. Increasing the discovery budget also causes many items to become Retained despite independently supported differences in success probability. A separate Qwen checkpoint study follows 95 items with confirmed improvements and finds a subsequent decline of about 12 percentage points, while performance remains above baseline.
The paper distinguishes observed success-set transitions, changes in success probability, and persistence across checkpoints.
Critical strengths
-
The separation of selection and validation responses is a meaningful strength. The paper tests whether fresh responses support the direction implied by an observed flip, rather than reusing the responses that selected the item. It distinguishes unresolved comparisons from evidence for the opposite direction and applies correction across both directions of all 151 selected Llama items. Table 2 provides the endpoint counts and effect sizes behind the strongest reversals. The budget analysis also gives a concrete illustration of how any-success labels can hide independently supported differences.
-
The Qwen decline has substantial supporting evidence. A separate response pool shows a similar decline, and item-level changes agree closely across pools, with a Spearman correlation of 0.947. The decline becomes smaller but remains negative after removing the largest contributors, and it persists when any single domain is removed. The authors also acknowledge that the unmatched reference comparison is descriptive and does not establish a causal training effect. These checks strengthen the narrower conclusion about the selected items.
Critical weaknesses
- The contribution remains fairly narrow despite the extensive analysis.
The paper provides careful measurements of a familiar problem: finite-sample success labels can be misleading, and any-success coverage can conceal differences in success probability. The held-out direction checks add useful evidence, but it remains unclear how much broader understanding follows from these particular cohorts.
The strongest reversal findings concern one Llama comparison. The additional MATH-500 and D2 analyses show no opposite-direction support under the common rules in Table 24. Their different settings mean that this does not contradict the main result, but the paper offers limited guidance on when the problem becomes consequential in practice.
Similarly, the Qwen measurements establish a decline on the selected items, but do not explain whether the early gains reflect temporary peaks or a more persistent change in performance. The authors appropriately limit their claims, and the unmatched reference cannot resolve this question. I would like a clearer explanation of which practical conclusions would change because of this study, beyond the general recommendation to use held-out responses and report uncertainty.
- The primary reversal evidence is difficult to verify fully.
Appendix J states that the primary W evidence includes counts and response-membership records but not full answers. This allows numerical analysis of the counts but prevents direct inspection of what the scorer captured in the nine corrected reversals. Table 42 shows that 5 to 46 adverse score changes out of 1,024 would remove Holm support for individual reversals. This does not establish scoring errors, but it makes a targeted response audit valuable.
The unrecovered GAD training details and incomplete execution comparability also limit reproducibility. The dependence simulations identify a vulnerability of the statistical analysis, but the available partitions cannot determine the extent of dependence in the actual responses. The reported uncertainty is explicitly conditional on fixed items, fixed exports, and the response model; it should not be interpreted as evidence of reproducibility across training runs.
There are also presentation and reference issues that should be corrected. I repeatedly had to consult Table 7 to distinguish response pools from item cohorts. The switch between conditional Beta means and fixed-denominator yields makes closely related results harder to follow. For example, the two definitions produce 78.95% and 79.24% for C95 at t1. A main-text explanation of the pools, one worked example, and a consistent primary score would help. Figure labels are too small, and the gray subscript intervals in Table 2 are difficult to read.
Two reference entries contain mismatched information. Agarwal et al. (2024a) links to arXiv:2406.12009, which is FinTruthQA, while the intended distillation paper appears to be listed separately as 2024b. Toneva et al. (2019a) uses the title of Goodfellow and colleagues’ paper, “An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks” (arXiv:1312.6211), while the actual Toneva paper appears as 2019b. The authors should identify the intended sources, correct the entries and associated citations, and check the remaining bibliography. These errors do not by themselves invalidate the numerical results, but they raise concerns about the manuscript’s verification.
Overall, I lean toward weak rejection. The held-out measurements are careful and useful, but the limited additional insight and incomplete verification of the primary reversal evidence weaken the case for acceptance. Clearer practical implications and directly inspectable evidence for the decisive examples could change my assessment.
Questions for authors
-
Can reviewers access the exact GAD export and the full primary W responses for the nine corrected reversals? Have these examples been checked manually to verify the scoring and clarify what behavior changed?
Please also clarify whether the Native/GAD endpoints and the D, V, and W pools were evaluated under consistent inference settings. Were the models self-hosted or accessed through an external provider, and were precision or quantization, inference engine and backend, prompt templates, and decoding settings matched? Table 6 reports BF16 and some runtime details, but it is unclear whether these apply consistently across all three pools.
If configurations differed, or the original responses cannot be recovered, could a targeted fresh evaluation under a common, documented setup reproduce the reversals while retaining the full responses and scorer outputs? This would help distinguish finite-sample label changes from differences introduced by the evaluation setup.
-
D contains 120 responses per endpoint outside the original K = 8 discovery prefix. What directions do these responses support for the 151 selected items, particularly the nine W reversals? Reporting their counts, effect sizes, and uncertainty would help establish whether the contrary directions are also visible within the original response source.
-
Could a selection-and-sampling model help explain the observed number and size of reversals using the actual discovery, validation, and multiple-testing procedure? A model fitted to W could provide descriptive context, provided the reuse of W and uncertainty in the estimated probabilities are acknowledged.
-
What practical conclusion should readers change after this study? For the Qwen result, what can the available checkpoints establish about temporary early gains versus a more persistent decline? I recognize that this distinction is not required to establish the observed fixed-cohort decrease, but it matters for the broader lesson about improvement stability.
Thank you for your feedback. In future work, we plan to develop a new method that builds on the findings of this paper and addresses the limitations identified in the evaluation. We will validate the method through additional experiments and compare it with relevant baselines.