1. The proposed local abduction does not actually localise faults
This is my main technical concern.
The name Local Abduction may suggest that the method identifies a locally incorrect part of a symbolic hypothesis and uses symbolic reasoning to repair it. However, Section 3.2 and Algorithm 2 show a different procedure. The search applies predefined editing actions at every eligible instruction position. It then executes the edited programs and checks whether their final outputs match the answer. There is no explicit analysis of which intermediate step is incorrect, no backward propagation of constraints through the execution, and no proof- or dependency-guided fault localisation.
In particular, Equation (6) constructs the neighbours by taking a union over all instruction positions and applicable edit actions. The beam is subsequently selected mainly by accumulated edit distance, with additional random candidates. The model likelihood is used after answer-consistent candidates have been found, rather than to identify the faulty program locations during search.
Consequently, local here means a small edit distance from the failed program. It does not mean that the method has identified the actual faulty part. These two notions are quite different.
I suggest that the authors state this distinction explicitly, and avoid implying a stronger diagnosis mechanism than is implemented. More importantly, they should compare against a fault-guided repair method. Useful alternatives include backward reasoning over intermediate results, dependency-based localisation, and algorithmic debugging when an execution or proof tree is available. The recent work Procedural Refinement by LLM-driven Algorithmic Debugging for ARC-AGI-2 is directly relevant to this discussion and should be cited and differentiated. Its task setting is different, so a direct benchmark comparison may require adaptation, but the difference in error diagnosis should be discussed carefully.
2. The local search has an important scalability and coverage limitation
Without fault localisation, the search needs to consider edits at many possible positions. If a program has n editable positions and roughly K alternatives per position, changing exactly k distinct positions can give an order of (kn)Kk candidates. Allowing up to r changed positions gives a neighbourhood of order ∑k=0r(kn)Kk in this simplified setting. This is combinatorial, although it is not generally O(n!). Insertions, deletions, repeated edits and dependencies can further change the search space.
LAST makes the search practical through a small radius (r=2), a restricted beam (B=256), random supplementation (R=64), and a three-second time limit. These are reasonable engineering choices, but they restrict coverage rather than solve the fault localisation problem. A correct program can be inside the edit neighbourhood and still be missed under the search budget. Alternatively, it may be outside the neighbourhood entirely.
The FinQA reachability analysis is informative: only 26.3% of the examined parseable failures are within two edits of their reference programs. This statistic does not exclude other valid repairs, but it shows why the underlying locality assumption deserves much more attention.
I would like to see repair success and verifier calls reported against program length, number of actual faulty positions, edit radius, and search budget. Please distinguish two cases: (i) a valid repair is unreachable with the permitted edits, and (ii) a valid repair is reachable but the search does not find it. Reporting only downstream accuracy does not reveal which limitation dominates. The authors should also compare with stronger search prioritisation that uses intermediate execution constraints rather than edit distance alone.
3. The novelty relative to existing abductive learning and program repair is not sufficiently established
The overall learning loop is familiar: propose a symbolic hypothesis, use background symbolic execution and the observed answer to correct it, then train the neural model on the corrected hypothesis. The paper appropriately cites Abductive Learning, Learning by Fixing, and recent work on abductive reflection. However, the distinctions are presented mostly at the level of task setting and representation.
The actual repair mechanism is a bounded enumeration of operator and operand edits, followed by answer checking and likelihood ranking. This seems considerably narrower than the more general term abductive inference may suggest. It does not induce new predicates, learn new operators, or infer a missing program structure outside the predefined edit neighbourhood.
I encourage the authors to provide a direct, technical comparison of hypothesis spaces, available constraints, fault identification, search operators, and computational complexity. Ideally, an adapted baseline based on existing weakly supervised program correction should also be included. If the main contribution is the use of bounded program repair to enlarge self-training supervision, that is a valid and clear positioning. It should not be presented as a general solution to abductive reasoning over failed hypotheses.
4. Answer consistency does not guarantee semantically correct supervision
Equation (3) accepts any edited solution satisfying S(z)=y. But several programs can produce the correct numerical answer for the wrong reason. The authors recognise this problem and use model likelihood to rank accepted repairs.
The concern is that likelihood is not a validity proof. Indeed, Appendix C reports a manual audit of 100 retained geometry repairs: 17% were judged spurious, while 28% matched the reference and 55% were valid alternatives. This is a useful and candid analysis, but it also shows that incorrect reasoning enters the training set despite answer verification and likelihood selection.
I suggest reporting semantic validity over more datasets and rounds, with confidence intervals and a clear annotation protocol. It would also be useful to measure whether spurious targets accumulate or reinforce each other over the six rounds. Where possible, evaluate repairs against intermediate geometric constraints, operand provenance, or additional independently sampled values, not only the final answer. A counterfactual test that changes the numerical inputs while preserving the problem structure could help distinguish genuine relations from accidental numerical agreement.
The paper should also be careful when interpreting exact reference-program match as logical correctness. A valid alternative derivation may differ from the reference; conversely, answer agreement alone is too weak. The authors are aware of the former issue, and the manual audit is a good start, but the distinction needs a more systematic evaluation.
5. More precise compute-matched and robust evaluations would strengthen the claims
The ablations show that local repair adds useful training targets. The comparisons with ENVISIONS and LAST-MS, the update-matched control, and the wall-clock analysis are all valuable. However, the main results still depend on several search and selection budgets: the number of candidate edits, symbolic verifier executions, language-model likelihood evaluations, and the time allowed per problem.
I suggest reporting the number of symbolic candidates tested per successful repair, successful repairs per unit of compute, and final test accuracy at matched total training and search time. Comparisons with extra neural sampling or repeated refinement under the same total budget would clarify whether structured local search is the more effective use of compute.
Finally, most primary results use one seed. The additional three-seed Geometry3K analysis is encouraging, but the stronger claims across all datasets would benefit from at least some further variability reporting. Also, standard LAST does not outperform ENVISIONS in the average 8B results in Table 1; the stronger result comes from LAST-MS under multi-sample exploration. The paper should maintain this distinction consistently.
We sincerely appreciate your important and valuable suggestions, as well as your positive assessment of our work.
A1. Hyperparameter selection
We selected the retention fraction ρ based on validation performance. Since the Geometry3K split provided by PGPS9K does not include a validation set, we used the GeoQA validation set for hyperparameter selection. Specifically, we evaluated \rho \in \\{25\\%, 50\\%, 75\\%, 100\\%\\} on the GeoQA validation set using models trained on Geometry3K. The results are as follows:
We selected ρ=25% as it achieved the highest validation accuracy among the evaluated settings.
To further understand this choice, we analyze the trade-off between supervision quantity and quality using reference-match precision, recall, and F1 on the Geometry3K training set. Among the first-round repair candidates, we select the highest-likelihood solution for each successfully repaired problem, obtaining 1,000 answer-consistent repair labels, of which 189 match the reference programs. After ranking these labels by likelihood, let n denote the number of retained labels and g the number matching the reference. We define precision as P=g/n, recall as R=g/189, and F1=2PR/(P+R). The results are as follows:
Increasing the retention fraction improves recall but reduces precision. Retaining the top 25% preserves approximately half of the reference-matching solutions while using only one-quarter of the repair labels, achieving the highest F1 among the evaluated settings. This analysis provides additional support for the choice of ρ=25%, highlighting its balance between supervision quantity and label quality.
A2. Performance comparison with LBF-style methods
Please refer to our response to this issue in G1.
A3. Robustness of search initialization
Following the random-start control in the paper, we add experiments with seeds 42/43/44 on Geometry3K and MathQA. The two groups complete six rounds of self-training starting either from the model's failed solutions or from random programs. The results are as follows:
On Geometry3K and MathQA, LAST outperforms random starting points for all three seeds, with average improvements of 2.60 and 5.54 percentage points, respectively. This indicates that using the model's failed solutions as search starting points yields consistent gains across seeds on different tasks.