acceptodds
Under review as a conference paper at ICLR 2027

Learning from Failed Hypotheses: Local Abduction for Neuro-Symbolic Self-Training

Abstract

Combining language models with symbolic solvers enables explicit and verifiable reasoning, but learning to construct symbolic solutions is hindered by the scarcity of annotations. Weakly supervised self-training addresses this challenge by learning from the model's own predictions verified against observed answers, yet new supervision still depends on generating a complete correct solution. We observe that failed solutions can retain useful structure, suggesting that supervision may be recovered before the model can solve a problem on its own. We propose Local Abductive Self-Training (LAST), which formulates this recovery as local abductive inference. Given a failed hypothesis, LAST searches its neighborhood for symbolic solutions consistent with the observed answer and uses model likelihood to select promising corrections. These corrections complement successful neural predictions and accumulate in trajectory memory to support subsequent learning. Across five geometry and numerical reasoning benchmarks, LAST improves average accuracy after six rounds by 7.84–10.69 and 4.83–12.40 percentage points over self-training baselines with 2B and 8B models, respectively. Further analyses show that LAST broadens supervision coverage while preserving solution quality, highlighting the value of failed hypotheses as a source of supervision.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Neuro-Symbolic Systems, Abductive Reasoning, Self-Training, Weakly Supervised Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_n5m7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The main problem this paper addresses is how, in neuro-symbolic self-training where only problems and answers are available, to obtain symbolic solutions for further training the model. It combines the model's own refinement with a local beam search that starts from the model's failed solution and looks for nearby solutions whose execution matches the known answer. The paper plugs this local edit-based search into the LLM self-training pipeline and reports improvements over self-training baselines on five geometry and numerical reasoning benchmarks.

Critical strengths
  1. The paper addresses a real problem in the neuro-symbolic field. The method is simple and effective, and the writing is fluent and clearly organized, making the paper easy to follow.
  2. The experiments are rigorous and fair, and the ablation studies are clean and comprehensive.
  3. The analysis shows that searching from failed solutions is far more efficient than searching from random starting points (the proportion of problems repaired in the first round is 13.46% vs. 4.25%), which supports the claim that failed solutions retain useful structure.
Critical weaknesses
  1. The paper does not explain how the hyperparameters were selected. The analysis of ρ\rho is conducted on the Geometry3K test set, and the value finally adopted is exactly the best one among those tested. A further explanation of the criteria used to select the optimal hyperparameters would be helpful.
  2. As I understand it, the main new component of the paper is the local abduction module, i.e., repairing the model's incorrect predictions under weak supervision and using the repaired results for subsequent training. This idea is fairly close to that of LBF (Hong et al., 2021). The paper does cite LBF, but it lacks a performance comparison between the proposed method and LBF-style methods on the arithmetic tasks.
Questions for authors
  1. Could the authors clarify the criteria used to select the hyperparameters?
  2. Could the authors add a performance comparison with LBF-style methods on the arithmetic tasks?
  3. In terms of final accuracy, random starting points (50.93%) and failed solutions (53.31%) differ by only about 2.4 points. Is this gap stable acros runs and on the other datasets?
3 comments

We sincerely appreciate your important and valuable suggestions, as well as your positive assessment of our work.

A1. Hyperparameter selection

We selected the retention fraction ρ\rho based on validation performance. Since the Geometry3K split provided by PGPS9K does not include a validation set, we used the GeoQA validation set for hyperparameter selection. Specifically, we evaluated \rho \in \\{25\\%, 50\\%, 75\\%, 100\\%\\} on the GeoQA validation set using models trained on Geometry3K. The results are as follows:

Retention fractionGeoQA validation accuracy (%)
25%29.53
50%25.50
75%27.92
100%28.99

We selected ρ=25%\rho=25\% as it achieved the highest validation accuracy among the evaluated settings.

To further understand this choice, we analyze the trade-off between supervision quantity and quality using reference-match precision, recall, and F1 on the Geometry3K training set. Among the first-round repair candidates, we select the highest-likelihood solution for each successfully repaired problem, obtaining 1,000 answer-consistent repair labels, of which 189 match the reference programs. After ranking these labels by likelihood, let nn denote the number of retained labels and gg the number matching the reference. We define precision as P=g/nP=g/n, recall as R=g/189R=g/189, and F1=2PR/(P+R)F_1=2PR/(P+R). The results are as follows:

Retention fractionReference-match precisionReference-match recallF1
25%38.00%50.26%43.28%
50%27.80%73.54%40.35%
75%22.40%88.89%35.78%
100%18.90%100.00%31.79%

Increasing the retention fraction improves recall but reduces precision. Retaining the top 25% preserves approximately half of the reference-matching solutions while using only one-quarter of the repair labels, achieving the highest F1 among the evaluated settings. This analysis provides additional support for the choice of ρ=25%\rho=25\%, highlighting its balance between supervision quantity and label quality.

A2. Performance comparison with LBF-style methods

Please refer to our response to this issue in G1.

A3. Robustness of search initialization

Following the random-start control in the paper, we add experiments with seeds 42/43/44 on Geometry3K and MathQA. The two groups complete six rounds of self-training starting either from the model's failed solutions or from random programs. The results are as follows:

DatasetSearch starting pointSeed 42Seed 43Seed 44Mean ± standard deviation
Geometry3KRandom program50.9348.0549.0749.35±1.46
Geometry3KModel's failed solution (LAST)53.3151.4451.1051.95±1.19
MathQARandom program43.9639.4242.7242.03±2.35
MathQAModel's failed solution (LAST)46.7048.2147.8047.57±0.78

On Geometry3K and MathQA, LAST outperforms random starting points for all three seeds, with average improvements of 2.60 and 5.54 percentage points, respectively. This indicates that using the model's failed solutions as search starting points yields consistent gains across seeds on different tasks.

Thank you for the additional experiments and clarifications. The validation results clarify how the retention fraction was selected, and the multi-seed results show consistent gains from starting with failed solutions on both datasets. These additions address my concerns, and I will maintain my positive rating.

Dear Reviewer n5m7,

Thank you very much for your response. We are delighted that we have been able to address your concerns, and we sincerely appreciate your continued support for our work!

Best regards,
The Authors

Reviewer_uhPmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LAST, a neuro-symbolic self-training framework that recovers supervision from failed symbolic solutions through local abductive search. The method uses symbolic execution to verify candidate corrections, model likelihood to select training targets, and trajectory memory for iterative learning. Experiments across five benchmarks demonstrate consistent improvements over existing self-training methods.

Critical strengths
  • The motivation is clear, and recovering useful supervision from failed predictions is an interesting direction.
  • The experiments cover multiple datasets and model sizes, with meaningful performance improvements and useful ablation studies.
Critical weaknesses
  1. The main approach combines edit-based search, symbolic verification, likelihood ranking, and iterative self-training, which are largely established techniques. The lack of direct comparisons with adapted symbolic repair methods makes it difficult to assess the distinct contribution of the proposed approach.

  2. Matching the final answer does not guarantee valid reasoning. The authors' own analysis identifies 17% of sampled repairs as spurious, raising concerns about the reliability of the generated training targets and the potential propagation of incorrect reasoning during self-training.

Questions for authors
  1. How does LAST compare with existing symbolic repair approaches under comparable computational budgets?
  2. Can the authors introduce stronger semantic verification or filtering to reduce spurious repairs?
  3. How well does the approach generalize to more complex symbolic representations requiring larger structural changes?
3 comments

We sincerely thank you for your valuable feedback, and we believe that this important feedback can improve the quality of our work.

A1. Compare with existing symbolic repair approaches

Please refer to our response to this issue in G1.

A2. Stronger semantic verification

Please refer to our response to this issue in G2.

A3. Generalization

To examine whether the self-training benefits of LAST transfer to different program representations, we add a Python-output experiment on FinQA. We evaluate the test set using Qwen3-VL-8B and the LAST and LAST (w/o Abduction) models after six rounds of self-training, changing the example programs in the evaluation prompt and the required output format to Python programs. The cross-representation generalization results are as follows:

ModelAnswer accuracy with Python output (%)
Qwen3-VL-8B65.81
LAST (w/o Abduction)60.83
LAST72.63

LAST achieves 72.63%, an improvement of 6.82 percentage points over the initial model, demonstrating the ability of the gains obtained from self-training to transfer to Python program representations. Error analysis finds that the accuracy decrease of LAST (w/o Abduction) relative to the initial model mainly arises from inconsistencies in output conventions for ratios and percentages, rather than from the reasoning structure.

We also analyze BPR's repair capabilities on longer programs and with larger modification ranges: among failed FinQA programs with 5–8 steps, BPR still repairs 29.08%. Of the successfully repaired programs, 60.98% pass the counterfactual test. Under the same 3-second time limit, increasing the edit radius from 2 to 3 raises overall repair coverage from 52.20% to 55.78%, and counterfactually consistent repair yield from 33.76% to 35.28%. These results provide additional evidence of LAST's scalability from three perspectives: cross-representation transfer, long-program repair, and repair with an expanded edit neighborhood.

Thanks for the detailed response and additional experiments. They help address some of my concerns, but I still have reservations about the novelty and reliability of the method. I'll keep my original rating.

Dear Reviewer uhPm,

Thank you very much for your response. We are glad that our response has helped address some of your concerns. We respect your assessment and sincerely appreciate your thoughtful and valuable feedback on our work!

Best regards,
The Authors

Reviewer_S4BNOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Local Abductive Self-Training (LAST) for learning to generate symbolic solutions from problems with answer-only supervision. When an LLM's initial solution and its refinement both fail, LAST searches a bounded edit neighbourhood around the failed program. Candidate programs are executed by a symbolic solver and checked against the known answer. The method selects answer-consistent candidates using length-normalised model likelihood, retains a subset as pseudo-labels, and incorporates them into trajectory memory for subsequent self-training.

The paper evaluates LAST on three geometry benchmarks and two numerical reasoning benchmarks, using 2B and 8B vision-language models. The experimental improvements are useful, and the authors include several relevant ablations. However, I have substantial concerns about the algorithmic novelty, the absence of fault localisation, and the reliability of the supervision obtained from answer consistency alone.

Critical strengths
  1. The paper considers a relevant weakness of self-training: a problem may provide no supervision when the model cannot generate a fully correct symbolic solution. Recovering information from partially correct failures is a reasonable objective.

  2. The experiments cover five datasets and two model sizes. The authors compare with STaR, iterative SFT, ENVISIONS and distillation-based methods. I appreciate the matched multi-sample LAST-MS comparison, and the ablation which removes only the local repair component.

  3. The paper does not rely only on final accuracy. It also examines repair reachability, supervision coverage, likelihood-based selection, additional training updates, wall-clock cost, and the semantic quality of a sample of repaired solutions. These analyses are helpful and should be retained.

Critical weaknesses

1. The proposed local abduction does not actually localise faults

This is my main technical concern.

The name Local Abduction may suggest that the method identifies a locally incorrect part of a symbolic hypothesis and uses symbolic reasoning to repair it. However, Section 3.2 and Algorithm 2 show a different procedure. The search applies predefined editing actions at every eligible instruction position. It then executes the edited programs and checks whether their final outputs match the answer. There is no explicit analysis of which intermediate step is incorrect, no backward propagation of constraints through the execution, and no proof- or dependency-guided fault localisation.

In particular, Equation (6) constructs the neighbours by taking a union over all instruction positions and applicable edit actions. The beam is subsequently selected mainly by accumulated edit distance, with additional random candidates. The model likelihood is used after answer-consistent candidates have been found, rather than to identify the faulty program locations during search.

Consequently, local here means a small edit distance from the failed program. It does not mean that the method has identified the actual faulty part. These two notions are quite different.

I suggest that the authors state this distinction explicitly, and avoid implying a stronger diagnosis mechanism than is implemented. More importantly, they should compare against a fault-guided repair method. Useful alternatives include backward reasoning over intermediate results, dependency-based localisation, and algorithmic debugging when an execution or proof tree is available. The recent work Procedural Refinement by LLM-driven Algorithmic Debugging for ARC-AGI-2 is directly relevant to this discussion and should be cited and differentiated. Its task setting is different, so a direct benchmark comparison may require adaptation, but the difference in error diagnosis should be discussed carefully.

2. The local search has an important scalability and coverage limitation

Without fault localisation, the search needs to consider edits at many possible positions. If a program has nn editable positions and roughly KK alternatives per position, changing exactly kk distinct positions can give an order of (nk)Kk\binom{n}{k}K^k candidates. Allowing up to rr changed positions gives a neighbourhood of order ∑k=0r(nk)Kk\sum_{k=0}^{r}\binom{n}{k}K^k in this simplified setting. This is combinatorial, although it is not generally O(n!)O(n!). Insertions, deletions, repeated edits and dependencies can further change the search space.

LAST makes the search practical through a small radius (r=2r=2), a restricted beam (B=256B=256), random supplementation (R=64R=64), and a three-second time limit. These are reasonable engineering choices, but they restrict coverage rather than solve the fault localisation problem. A correct program can be inside the edit neighbourhood and still be missed under the search budget. Alternatively, it may be outside the neighbourhood entirely.

The FinQA reachability analysis is informative: only 26.3% of the examined parseable failures are within two edits of their reference programs. This statistic does not exclude other valid repairs, but it shows why the underlying locality assumption deserves much more attention.

I would like to see repair success and verifier calls reported against program length, number of actual faulty positions, edit radius, and search budget. Please distinguish two cases: (i) a valid repair is unreachable with the permitted edits, and (ii) a valid repair is reachable but the search does not find it. Reporting only downstream accuracy does not reveal which limitation dominates. The authors should also compare with stronger search prioritisation that uses intermediate execution constraints rather than edit distance alone.

3. The novelty relative to existing abductive learning and program repair is not sufficiently established

The overall learning loop is familiar: propose a symbolic hypothesis, use background symbolic execution and the observed answer to correct it, then train the neural model on the corrected hypothesis. The paper appropriately cites Abductive Learning, Learning by Fixing, and recent work on abductive reflection. However, the distinctions are presented mostly at the level of task setting and representation.

The actual repair mechanism is a bounded enumeration of operator and operand edits, followed by answer checking and likelihood ranking. This seems considerably narrower than the more general term abductive inference may suggest. It does not induce new predicates, learn new operators, or infer a missing program structure outside the predefined edit neighbourhood.

I encourage the authors to provide a direct, technical comparison of hypothesis spaces, available constraints, fault identification, search operators, and computational complexity. Ideally, an adapted baseline based on existing weakly supervised program correction should also be included. If the main contribution is the use of bounded program repair to enlarge self-training supervision, that is a valid and clear positioning. It should not be presented as a general solution to abductive reasoning over failed hypotheses.

4. Answer consistency does not guarantee semantically correct supervision

Equation (3) accepts any edited solution satisfying S(z)=yS(z)=y. But several programs can produce the correct numerical answer for the wrong reason. The authors recognise this problem and use model likelihood to rank accepted repairs.

The concern is that likelihood is not a validity proof. Indeed, Appendix C reports a manual audit of 100 retained geometry repairs: 17% were judged spurious, while 28% matched the reference and 55% were valid alternatives. This is a useful and candid analysis, but it also shows that incorrect reasoning enters the training set despite answer verification and likelihood selection.

I suggest reporting semantic validity over more datasets and rounds, with confidence intervals and a clear annotation protocol. It would also be useful to measure whether spurious targets accumulate or reinforce each other over the six rounds. Where possible, evaluate repairs against intermediate geometric constraints, operand provenance, or additional independently sampled values, not only the final answer. A counterfactual test that changes the numerical inputs while preserving the problem structure could help distinguish genuine relations from accidental numerical agreement.

The paper should also be careful when interpreting exact reference-program match as logical correctness. A valid alternative derivation may differ from the reference; conversely, answer agreement alone is too weak. The authors are aware of the former issue, and the manual audit is a good start, but the distinction needs a more systematic evaluation.

5. More precise compute-matched and robust evaluations would strengthen the claims

The ablations show that local repair adds useful training targets. The comparisons with ENVISIONS and LAST-MS, the update-matched control, and the wall-clock analysis are all valuable. However, the main results still depend on several search and selection budgets: the number of candidate edits, symbolic verifier executions, language-model likelihood evaluations, and the time allowed per problem.

I suggest reporting the number of symbolic candidates tested per successful repair, successful repairs per unit of compute, and final test accuracy at matched total training and search time. Comparisons with extra neural sampling or repeated refinement under the same total budget would clarify whether structured local search is the more effective use of compute.

Finally, most primary results use one seed. The additional three-seed Geometry3K analysis is encouraging, but the stronger claims across all datasets would benefit from at least some further variability reporting. Also, standard LAST does not outperform ENVISIONS in the average 8B results in Table 1; the stronger result comes from LAST-MS under multi-sample exploration. The paper should maintain this distinction consistently.

Questions for authors
  1. Fault localisation. How does LAST decide which instruction is likely to be wrong? If it does not attempt this, could the authors explicitly describe their approach as bounded answer-guided edit search rather than fault-localising abduction? Can an execution-guided localisation baseline be evaluated?

  2. Search guarantees and scaling. What fraction of unsuccessful repairs are outside the edit neighbourhood, and what fraction are inside but missed because of the beam or time budget? How do verifier calls, repair coverage, and repair precision change with program length and the number of necessary edits?

  3. Relation to prior work. What precise technical problem is solved beyond earlier abductive learning, Learning by Fixing, and algorithmic debugging approaches? Please discuss Procedural Refinement by LLM-driven Algorithmic Debugging for ARC-AGI-2, especially the role of structured diagnosis versus blind local edits.

  4. Semantic correctness. Could the authors evaluate corrected programs on perturbed numerical instances or independently checked intermediate constraints? How much of the training improvement comes from semantically valid corrections, as opposed to answer-consistent but spurious ones?

  5. Role of the model likelihood. Does the likelihood score reliably favour semantically correct repairs on problems with many answer-consistent programs? A ranking-quality analysis (for example, semantic validity among the top-kk candidates) would be useful.

  6. Compute efficiency. Under the same total compute budget, how does LAST compare with more model samples, more refinement attempts, or a stronger symbolic repair baseline? Please include the search and likelihood-scoring overhead, not only the number of extra training labels.

8 comments

We sincerely appreciate your professional and thoughtful review, and we believe that this valuable feedback will be very helpful in further improving the quality of our work.

A1. Fault localisation

We greatly appreciate the reviewer's detailed understanding of and reflections on LAST's methodology.

In line 92 of the paper, we explain the meaning of local abduction: searching for an answer-consistent correction within a bounded neighborhood of the failed hypothesis. The method does not include a fault-localization mechanism.

The term local abduction is mainly used to convey our modeling perspective on the program repair problem. Our motivation and reasons for using this name are as follows:

  1. Abduction: The concept of abduction [1] describes the process of finding possible logical explanations for an observed answer given a knowledge base (Sec. 3.2; [2], Sec. 2.1; [3]). Our program repair problem concerns how to find symbolic solutions consistent with a known answer in order to recover supervision. We therefore use the concept of abduction to reflect our problem-modeling perspective.
  2. Local: We use the term local to indicate that we make full use of the model's existing predicted structure and search within a bounded edit neighborhood of its failed hypothesis. The "neighborhood" illustration in Fig. 1(a) conveys the intended meaning of local.

However, we recognize that this name may cause misunderstanding, and we thank the reviewer for pointing this out. We will make two changes:

  1. We will no longer use the single term local abduction to refer interchangeably to both the modeling perspective and the solution technique. In fact, our method reflects the abductive perspective only at the modeling stage, whereas the subsequent program repair algorithm is a specific solution technique for the modeled problem and should not also fall under the scope of the term abduction.

    We will use abduction only to refer to the problem-modeling perspective we follow, and will explicitly state that local refers to a small edit distance. Meanwhile, we will name our proposed symbolic program repair algorithm Bounded Program Repair (BPR), to avoid leading readers to regard it as part of general abductive inference.

  2. We will explicitly discuss the technical differences between BPR and repair algorithms based on constraint propagation or fault localization in the Related Work section, and clarify the positioning of our work. See A3 for a more detailed discussion.

In G1, we compare BPR with LBF, an execution-guided candidate value localisation method that derives feasible substitutions through backward constraint propagation. Please see G1 for details.

[1] Abductive cognition: The epistemological and eco-cognitive dimensions of hypothetical reasoning.

[2] Bridging Machine Learning and Logical Reasoning by Abductive Learning.

[3] Curriculum Abductive Learning.

A2. Search guarantees and scaling

We thank the reviewer for the analysis of the algorithm's complexity.

We conduct an analysis of search failures and budgets on the same 2,180 failed model Exploration programs from FinQA as in Fig. 1(b) of the paper.

On these problems, our BPR algorithm finds answer-consistent solutions for 1,138 problems (52.20%). The failure causes for the remaining 1,042 problems are as follows:

Cause of failureNumber of problemsPercentage of failed samples
No repair program exists within radius 273670.63%
A repair program exists within radius 2 but the search is truncated by parameter limits30629.37%

The absence of an answer-consistent repair within radius 2 was established by exhaustive search under the same edit actions, without beam pruning or the search-time limit.

In addition, we analyze the search results by program length, minimum number of edits, and edit radius. The minimum number of edits d∗d^* is defined as the minimum number of edits required to transform the failed program into any answer-consistent program. The results are as follows:

Program lengthNumber of problemsAverage verifier callsRepair coverageCounterfactual consistency rate
1 step8171,22354.59%87.89%
2 steps8306,79356.39%51.92%
3 steps31516,69149.21%40.65%
4 steps7726,51236.36%46.43%
5–8 steps14123,33929.08%60.98%
d∗d^*Number of problemsAverage verifier callsRepair coverageCounterfactual consistency rate
18537,02685.81%65.03%
25918,36768.70%64.04%
No feasible repair within radius 27368,5440.00%—
Radius / nominal time budgetRepaired problemsCoverageCounterfactual consistency rateConsistent repair yieldAverage verifier callsAverage CPU seconds
1 / 3 s65730.14%69.56%20.96%1290.013
2 / 3 s1,13852.20%64.67%33.76%7,9020.899
3 / 3 s1,21655.78%63.24%35.28%18,3771.939

These results show that, as programs become longer or require more edits, verification cost generally increases and repair coverage generally decreases, whereas counterfactual consistency does not decrease monotonically. As the minimum number of edits increases, the coverage of successful repairs decreases, but the counterfactual consistency rates remain similar. Increasing the edit radius from 2 to 3 improves coverage by 3.58 percentage points and consistent repair yield by 1.52 percentage points, but increases average CPU time to approximately 2.16 times its original value, illustrating the benefits and costs of expanding the search.

We acknowledge that the absence of explicit fault localization requires the search to consider multiple potential edit positions. In practice, empirical parameter choices are therefore needed to balance repair gains and computational cost. The above analysis demonstrates the benefits of the current configuration while also making its limitations explicit.

Nevertheless, the comparison with LBF shows that the current search strategy can still achieve competitive repair performance. Moreover, this paper focuses on recovering additional supervision from failed programs at a relatively manageable cost and improving the model's subsequent independent problem-solving ability through self-training. Thus, our main contribution concerns the role of symbolic repair in self-training, rather than treating the performance optimization of the program repair algorithm itself as an independent research objective. We provide a more detailed discussion in A3.

A3. Relation to prior work

We sincerely thank the reviewer for the feedback and the recommended related literature.

Abductive Learning provides a general modeling framework for searching for candidate hypotheses using background knowledge and observed outcomes. LAST, LBF, and ABPR [1] all embody related modeling ideas, but adopt different solution strategies.

BPR in LAST follows a forward Propose-and-Verify approach: it generates candidates by applying edit actions at each editable position of a failed program, and then uses a symbolic solver to verify their answer consistency. The search proceeds layer by layer within a bounded edit neighborhood using breadth-first beam search.

LBF employs a backward repair strategy driven by the target answer. By representing the program as a tree and using the execution values and constraints of its subtrees, it propagates the final answer backward along the expression tree into target values for intermediate nodes, thereby finding feasible edit positions and symbol substitutions.

ABPR [1] emphasizes the importance of intermediate semantic analysis for program repair. It constructs structured derivation trees through symbolic execution and uses an LLM to judge the semantic correctness of intermediate predicate instances, thereby locating suspicious rules and generating targeted modifications. This design combines final-outcome feedback with semantic judgments of intermediate derivations, providing finer-grained guidance for fault localization and repair.

Importantly, the motivation behind BPR is closely related to the more general symbolic program structures considered in this paper (Sec. 2.1). Unlike LBF, which exploits the specific structure of arithmetic expression trees for backward constraint propagation, the symbolic programs considered here may involve complex constraint coupling between instructions and intermediate variables, and the overall answer constraint may not be decomposable into locally invertible constraints at individual nodes. We therefore adopt the Propose-and-Verify design, which allows the repair process to avoid relying on a particular constraint decomposition structure, thereby supporting supervision recovery across different structures through a unified search mechanism.

We further attempt to introduce an execution-based candidate value localization mechanism into BPR (BPR-CL) to investigate its effects on search efficiency and repair performance. Specifically, for operand replacement at the final layer of the search tree, suppose the original program corresponds to a system of constraint equations containing mm unknown variables, and there are nn legal candidate values at the position to be replaced. The original BPR enumerates the legal candidates one by one and solves the corresponding system of mm equations to verify answer consistency, potentially requiring up to nn solves. In contrast, BPR-CL treats the operand to be replaced as an additional unknown and introduces the known-answer constraint, constructing an augmented system with m+1m+1 unknown variables (whose degree of nonlinearity may increase). By solving this augmented constraint problem, we can derive replacement values satisfying the answer constraint, thereby reducing the need to solve and verify candidate programs.

We compare BPR and BPR-CL on the first-round predicted refinement-solution sets from LAST on Geometry3K, FinQA, and MathQA, using the same search settings as in the paper. The results are shown below:

DatasetMethodCoverage (%)Counterfactual consistency rate (%)Consistent repair yield (%)Average CPU time (seconds/problem)
Geometry3KBPR13.4662.008.341.464
Geometry3KBPR-CL13.1264.318.441.464
FinQABPR47.8064.2430.710.542
FinQABPR-CL48.0664.2530.880.187
MathQABPR32.3714.934.831.076
MathQABPR-CL32.5614.844.830.685

We observe that the consistent repair yield remains essentially unchanged or improves only slightly on all three datasets. Average CPU time is essentially unchanged on Geometry3K, and decreases by approximately 0.355 and 0.391 seconds/problem on FinQA and MathQA, respectively. These results suggest that, constraint-guided candidate localization mainly reduces verification cost rather than substantially increasing high-quality supervision.

Comments on the paper

2 comments

We would like to express our sincere gratitude for the reviewers' valuable feedback, which is very helpful for further improving the paper.

G1. Comparison with LBF

To address the reviewers' concerns regarding symbolic repair baselines (S4BN, Q1; uhPm, Q1; n5m7, Q2), we have added two comparisons with LBF [1]: program repair performance under the same inputs and time limits, and self-training performance after integrating LBF into LAST.

LBF can repair failed programs representable as expression trees. It derives intermediate constraints backward from the known answer and attempts a repair that changes one symbol (1-FIX); if this fails, it applies a random perturbation and tries again. It guides the search using a probability table of candidate symbols at each position. Table 1 sets the initial probabilities of candidate symbols to a uniform distribution (LBF-r), while Table 2 further uses model-probability guidance (LBF-m).

We name the program repair algorithm proposed in our paper Bounded Program Repair (BPR). The scope of this name is restricted to the symbolic search process that takes a failed program as input and generates answer-consistent candidate programs; it does not include likelihood-based selection.

We evaluate the two repair algorithms on the failed programs entering the repair module in the first round of LAST. Let NN denote the number of input problems, SS the number of successful repairs, and CC the number of those repairs that pass the counterfactual test. We define coverage as S/NS/N, the counterfactual consistency rate as C/SC/S, and the counterfactually consistent repair yield as their product, C/NC/N.

The counterfactual test is conducted as follows: for each candidate program, we apply random relative perturbations of ±10% to the perturbable input quantities provided by the problem, and compare the outputs of the candidate and reference programs under the same perturbed inputs. A candidate passes if the outputs agree within the specified numerical tolerance (1e-4) for all 10 sets of perturbed inputs.

The experimental results are shown in the following Table 1:

DatasetSearch methodCoverage (%)Counterfactual consistency rate (%)Consistent repair yield (%)CPU seconds/problem
FinQABPR47.8064.2430.710.542
FinQALBF-r57.1649.8528.490.731
FinQA (tree-structured subset)BPR57.3163.8936.610.507
FinQA (tree-structured subset)LBF-r73.9049.8536.840.945
MathQABPR32.3714.934.831.076
MathQALBF-r21.528.301.790.050
MathQA (tree-structured subset)BPR75.9016.6212.620.495
MathQA (tree-structured subset)LBF-r97.618.308.100.219

LBF achieves higher coverage on tree-structured subsets by exploiting tree structure for pruning, but this does not necessarily yield more counterfactually consistent supervision. On FinQA tree-structured subset, both methods achieve similar consistent repair yields; on MathQA tree-structured subset, BPR yields more consistent repairs, while LBF is faster. These results highlight a trade-off between coverage, quality, and cost. On the full datasets, BPR achieves higher counterfactual consistency rates and consistent repair yields.

We believe this difference may be related to the extent to which the search preserves the model's prior: BPR concentrates its budget in a local neighborhood of the original prediction, whereas LBF's random walk may accumulate more symbol substitutions, making it more likely to find solutions that match the answer but fail to preserve the correct input dependencies.

On FinQA, we further follow the model-probability-guided approach of the original LBF method: the original paper obtains the probability table from a tree prediction model, while here we construct a fixed probability table using the current LLM's conditional probabilities for candidate symbols under the prefix of the original failed program. The following Table 2 compares uniform probabilities with model probabilities:

Coverage (%)Counterfactual consistency rate (%)Consistent repair yield (%)CPU seconds/problem
LBF-r57.1649.8528.490.731
LBF-m33.4972.6024.311.380

As shown, model guidance increases the counterfactual consistency rate of successful repairs but reduces coverage, without improving the final consistent repair yield. One possible explanation is that the fixed probability table biases the search toward high-probability candidates while reducing opportunities to explore low-probability but feasible repairs. In addition, constructing the model probability tables for the repair of this dataset incurred approximately 2.04 additional hours of wall-clock time.

Considering the observed performance of the different LBF variants, and to maintain consistent resource budgets for program repair, we select LBF-r as a replacement for BPR while keeping all other modules unchanged. We compare LAST (LBF-r) with the original LAST after six iterations:

DatasetLASTLAST (LBF-r)
FinQA75.6775.67
MathQA46.7044.09

The two methods perform equally on FinQA, while the original BPR-based version is 2.61 percentage points higher on MathQA. Notably, BPR can also handle more general program structures than LBF (Sec. 2.1), such as the PGPS constraint language.

These comparisons indicate that, in self-training under weak answer supervision, stronger answer-fitting ability does not necessarily produce more reliable supervision. Rather than simply increasing search coverage, preserving the model's reasoning prior, limiting excessive deviations, and balancing repair quality with applicability may be more helpful for translating symbolic repair into improvements in the model's subsequent independent problem-solving ability. BPR achieves a competitive trade-off in these respects.

[1] Learning by fixing: Solving math word problems with weak supervision.

G2. Reliability of recovered supervision

To address the reviewers' concerns about the reliability of repaired labels and stronger filtering mechanisms (S4BN, Q4; uhPm, Q2), we have added counterfactual tests across training rounds and conducted a diagnostic filtering experiment to examine their impact on self-training.

We conduct counterfactual tests on the newly added symbolic repair labels retained after likelihood-based selection in rounds R1–R6 of LAST on Geometry3K and FinQA. We use this test to measure structural consistency under changes in the inputs. The results are as follows:

RoundGeometry3KFinQA
R186.80%85.87%
R286.94%80.79%
R376.95%79.28%
R473.10%58.97%
R575.33%70.49%
R666.67%52.17%
All six rounds79.46%77.46%

From these results, the cumulative pass rates over six rounds are 79.46% and 77.46% for Geometry3K and FinQA, respectively. The pass rates generally decline as the number of training rounds increases. Nevertheless, most newly added symbolic repair labels pass the counterfactual test, and the rate exceeds 50% in every round.

We further investigate whether introducing a stronger filtering mechanism can improve self-training gains on Geometry3K. We conduct two intervention-based diagnostic experiments:

  • LAST (counterfactual filtering): After likelihood-based selection in each round, remove newly added symbolic repair labels judged inconsistent by the counterfactual test.
  • LAST (random deletion): After likelihood-based selection in each round, randomly remove the same proportion of labels as is removed in that round by counterfactual filtering.

We emphasize that counterfactual filtering is an oracle-assisted diagnostic experiment, as it uses reference programs to assess the consistency of repaired labels. The results after six iterations are as follows:

MethodValue accuracy (%)
LAST (w/o Abduction)45.50
LAST53.31
LAST (counterfactual filtering)52.12
LAST (random deletion)50.93

The counterfactual-filtering group exceeds the random deletion group with the same deletion proportion by 1.19 percentage points, and remains 6.62 points above the group without symbolic repair, indicating that targeted filtering preserves more of the training gains in this experiment. However, its result is 1.19 points lower than standard LAST, suggesting a trade-off between consistency filtering and the quantity of supervision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.