- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article investigates whether modifying the training code of large language models can bring benefits beyond hyperparameter optimization. The author proposes a reachability analysis framework, which divides modifications into four categories: unchanged behavior, re-expression of existing parameters, reproducible by a pre-declared extended search space, and beyond this space. The author also builds a SelfTune platform to evaluate the effect of modifications through modification validation checks, independent program parameter adjustment, and comparison of parent and child programs under the same configuration. The author conducted editing corpus auditing, same-budget comparison, and coverage matching experiments on the Pythia model and ARC-Easy, RACE, and OpenBookQA tasks. About 76% of the 332 major edits were classified as parameter-level changes, and the significant advantage of program search over parameter adjustment control was not established under the tested settings. The main contribution of this paper is to provide an evaluation framework based on actual implementation evidence for distinguishing search space expansion from additional benefits of program search.
Critical strengths
- The authors propose a meaningful evaluation framework to assess the reproducibility, validity, and independent data gains of the modifications.
- The experiments analyze editing types, archival inclusion, execution variances, and configuration interactions, while also addressing data coverage and statistical limitations.
Critical weaknesses
-
The core ratio lacks item-by-item verification: 76% comes from type mapping across cohorts, the impact of implementation differences on this ratio is unclear.
-
Early pairwise comparisons are biased: The historical best results of the parent program are affected by selection bias and noise, which limits the judgment of the effect of modifications.
Questions for authors
-
Section 3 and Appendix B.4 state that a fixed four-anchor-point approach is used, whereas Appendix B.3 and Table 6 refer to the use of the four best configurations from the parent program. Why is this the case?
-
Is there any code or source documentation supporting implementation consistency across cohorts? What does the 76% figure represent when considering only edits with verifiable correspondences?
-
Are there independent parent-child re-evaluation results? If so, how do the paired labels from historical score comparisons differ from those obtained through independent re-evaluation?
-
The significance criteria in Section 5 are inconsistent with the results in Table 9. Please clarify whether this is a typographical error or due to different testing methods, and indicate whether it affects the relevant conclusions.
Thanks for the response. However, even if the ability of the agent with Λ+ was tested on new edits, it is still reliant on the model's output on other data. Thus, the definition of Λ+ differs when summarized from a corpus of size 100 and 10000. With the exact same behavior of the agent on the new 143 edits, R2/R3 edit rate can vary wildly; if the R3 edit rate is low, this does not necessarily mean that the agent does not know how to perform a novel edit - it may be just that the corpus is too big. Overall, I think my question is: can the R2/R3 rate be interpreted beyond a specific choice of corpus and test set? If not, then a low R3 edit rate may not indicate "agents cannot often propose R3 edits" even on another pair of corpus and test set of the same task domain.