acceptodds
Under review as a conference paper at ICLR 2027

VeriLoop Structural Repair: Quantifying Progress Without Regression in Code Repair

Abstract

Test-based program repair can hide damage to previously correct behavior when it reports only aggregate test success. We introduce VSR, which constructs training and evaluation data from protected-obligation rank vectors and the non-regression relation . The dataset contains 40 program families, 988 tasks, 2,297 states, and 27,046 executed candidates and passes 228,473 independent checks. Base/SFT comparisons at both 4B and 9B show that supervised repair data improves closed-loop completion. Construction ablations show that coupling-aware sampling exceeds equal-size random sampling at both model scales and on all three test distributions, while the full SFT mixture similarly exceeds COMPLETE-only in every matched comparison. Sequential 9B SFT-to-DPO raises completion by , , and on ID, OOD-family, and OOD-, respectively. In the DPO error-type ablations, removing either aggregate-trap or protected-regression examples causes a comparatively large drop in completion and usually lowers NRIR or increases calls, highlighting their roles in resisting aggregate-score shortcuts and preserving previously correct behavior.

Primary area:
datasets and benchmarks
Keywords:
Non-regression Code Repair, Automated Program Repair, Code Large Language Models, Dataset Construction, Execution-Based Verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eQ34OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VLS (VeriLoop Structural Repair), a dataset generation pipeline. VLS performs fault injection into small seed Python programs. The dataset then contains "repair records" which are going in the opposite direction, fixing the injected faults. What is different and novel here is that each program variant is checked against a set of "obligations", i.e., input-output pairs. The dataset is carefully selected to avoid "aggregate traps", where the total number of failed obligations decreases but the improvements in some obligations mask failures in others. VLS can also control for the number of obligations in each step, as well as "coupling", which measures effects across multiple fault injection sites and obligations. SFT generates positive samples for SFT, as well as purposely undesirable samples for DPO. Furthermore, VLS generates separate splits for ID and OOD experiments, and furthermore, OOD both with respect to the seed Python programs and the faults injected into them. The results demonstrate that VLS improves fine-tuning on both 4b and 9b models. The SFT portion yields the bigger gains. With DPO, the gains are bigger on OOD than on ID test data.

Critical strengths

S1. The paper reports results with both SFT and DPO.

S2. VLS directly avoids aggregate traps and helps control coupling.

S3. The evaluation separates ID from OOD results.

Critical weaknesses

W1. The paper uses a small number of small seed code samples and synthetic fault injection. While this may be okay for the training set, it is insufficient for evaluation. The paper could be stronger if it included results on a separate established benchmark such as LiveCodeBench.

W2. The paper has an impoverished related-work discussion. It only cites a small number of references, excluding some closely-related efforts. For instance, it would do well to also cite "Establishing Best Practices in Building Rigorous Agentic Benchmarks" (NeurIPS'25) and "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code" (ICLR'25). It would also be useful to add a citation for the base Qwopus models.

W3. The writing is unnecessarily dense. This could be mitigated by stating up-front the format of the generated dataset. Even better would be some short concrete examples.

Questions for authors

Q1. How well do your fine-tuned models perform on independent datasets such as LiveCodeBench?

Q2. You state that VLS starts from "40 short Python families". How short are they? Can you give a couple of representative examples?

Q3. The paper clearly defines "rank", but later also introduces the term "grade"; can you please provide a clear definition of "grade" and motivate it in comparison to "rank"?

Reviewer_PtXQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VeriLoop Structural Repair (VSR), a framework for constructing code-repair supervision that distinguishes improvement from regression on protected behavioral obligations. A candidate is admissible only if it repairs at least one failing obligation without breaking any passing obligation. VSR combines coupling-aware obligation sampling, hindsight-aligned diagnostics, safe intermediate repairs, and structured preference pairs. Experiments with 4B and 9B models show higher completion after supervised fine-tuning, with further gains from DPO and the proposed data-construction choices. Evaluation covers in-distribution tasks, unseen program families, and higher-coupling tasks generated within the same synthetic framework.

Critical strengths
  1. The paper addresses an important repair objective by explicitly separating newly corrected behavior from regressions in previously passing obligations.
  2. Experiments at two model scales show improved completion after supervised training, supported by targeted data-construction ablations.
Critical weaknesses
  1. The evaluation protocol does not clearly separate learned non-regression from verifier enforcement. Sections 5.2–5.3 state that all variants use the same acceptance rule, but do not specify which obligations the evaluator accesses or what happens after rejection. If hidden obligations control candidate acceptance and subsequent repair rounds, completion measures a model operating with a privileged verifier. This still permits comparisons under a common verifier, but does not establish that the model independently avoids regressions. Equation 11 defines NRIR, yet its denominator and aggregation across candidates, tasks, and seeds remain unclear. Reporting candidate-level regression rates and results with acceptance based only on visible tests would clarify the learned behavior. Transfer also remains untested outside the synthetic generator, despite holding out program families.

  2. The distinctive training signal is not sufficiently isolated, and uncertainty in the incremental gains is unreported. Native-model and component-removal comparisons do not replace a matched DPO baseline using aggregate pass-count preferences on the same candidate pool. Such a control would help distinguish the protected-obligation signal from the benefit of additional in-domain training. The random-sampling control also needs clarification on whether it preserves the detectability floor. Results are means over three seeds without per-seed values or uncertainty intervals, despite test subsets containing only 84–96 tasks. The 9B ID DPO gain corresponds to approximately one additional solved task per run on average. Moreover, removing equal-rank negatives produces a 9B NRIR decrease close to those from removing aggregate-trap or protected-regression negatives. The evidence therefore supports more qualified claims about the relative importance of individual negative types.

Questions for authors
  1. Which obligations govern acceptance during evaluation, what feedback follows rejection, and how is NRIR aggregated over generated candidates?
  2. How does VSR compare with DPO using aggregate pass-count preferences under matched training conditions, and does the random sampler retain the detectability floor?
  3. Can you report per-seed results and paired uncertainty estimates for the SFT-to-DPO and component-ablation gains at both model scales?
Reviewer_EuyDOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces method for generating dataset for program repair with non-regression progress. In the dataset structure the authors proposed, there are some positive patches with non-regression repair and some negative pairs with regression repair which are built upon a set of artificial python program (which is the same for both training, validating and testing set). They evaluate their training (or fine-tuning) performance for two small LLMs on their datasets and the results show that they have improvement on both supervised fine-tuning and direct preference optimization.

Critical strengths
  1. The idea is interesting and the methodology is intuitively correct.
  2. The ablation study shows that fine-tuning or optimized after the built datasets can provide improvement on program repair tasks.
  3. The authors provide their built artificial datasets and promise to publish them.
Critical weaknesses

The writing of this paper is not deeply based on the analysis of the problem. The introduction and related work describe high-level ideas without motivating the problem via domain-specific analysis. It reads like a solution without problem analysis. The authors' insight within the problem domain, what are regression bugs, how traditional methods fix them, how to translate the domain-specific understanding into this solution is important for a research paper. In addition, the paper only have very little related work discussed, while there is a large body of work in related areas in software engineering.

The idea is interesting but relatively simple. I feel the formalization in Section 3 is an overkill. We do not gain much by going to the level of abstraction, but it increases the difficulty of understanding. It is better to put more effort in problem analysis than unnecessary formalization to make the paper's technical content deeper.

The technical sessions are presented as a series of actions taken by the approach. It will be more interesting to see the discussion of design choices, problem analysis, etc. In other words, the thinking process behind the solutions. It will make the paper more interesting and easier to read.

Questions for authors

What is the main technical insight from the problem domain that leads to the key design of this paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.