- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces VLS (VeriLoop Structural Repair), a dataset generation pipeline. VLS performs fault injection into small seed Python programs. The dataset then contains "repair records" which are going in the opposite direction, fixing the injected faults. What is different and novel here is that each program variant is checked against a set of "obligations", i.e., input-output pairs. The dataset is carefully selected to avoid "aggregate traps", where the total number of failed obligations decreases but the improvements in some obligations mask failures in others. VLS can also control for the number of obligations in each step, as well as "coupling", which measures effects across multiple fault injection sites and obligations. SFT generates positive samples for SFT, as well as purposely undesirable samples for DPO. Furthermore, VLS generates separate splits for ID and OOD experiments, and furthermore, OOD both with respect to the seed Python programs and the faults injected into them. The results demonstrate that VLS improves fine-tuning on both 4b and 9b models. The SFT portion yields the bigger gains. With DPO, the gains are bigger on OOD than on ID test data.
Critical strengths
S1. The paper reports results with both SFT and DPO.
S2. VLS directly avoids aggregate traps and helps control coupling.
S3. The evaluation separates ID from OOD results.
Critical weaknesses
W1. The paper uses a small number of small seed code samples and synthetic fault injection. While this may be okay for the training set, it is insufficient for evaluation. The paper could be stronger if it included results on a separate established benchmark such as LiveCodeBench.
W2. The paper has an impoverished related-work discussion. It only cites a small number of references, excluding some closely-related efforts. For instance, it would do well to also cite "Establishing Best Practices in Building Rigorous Agentic Benchmarks" (NeurIPS'25) and "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code" (ICLR'25). It would also be useful to add a citation for the base Qwopus models.
W3. The writing is unnecessarily dense. This could be mitigated by stating up-front the format of the generated dataset. Even better would be some short concrete examples.
Questions for authors
Q1. How well do your fine-tuned models perform on independent datasets such as LiveCodeBench?
Q2. You state that VLS starts from "40 short Python families". How short are they? Can you give a couple of representative examples?
Q3. The paper clearly defines "rank", but later also introduces the term "grade"; can you please provide a clear definition of "grade" and motivate it in comparison to "rank"?