acceptodds
Under review as a conference paper at ICLR 2027

Keep or Repair? Certified Acceptance for Conservative Query Repair

Abstract

Language-model repairers are a natural add-on to knowledge base question answering (KBQA) parsers, but once the parser is accurate, repair is governed by the damage it does rather than the errors it fixes. When upstream accuracy reaches 92%, as on our KQA Pro development cohort, even a small fraction of wrongly edited correct answers can outweigh gains on incorrect questions. We formalize conservative query repair and separate what to propose from what to accept. Our certified acceptance gate accepts a proposed query only when the repairer's own KEEP-sequence probability κ on it exceeds a threshold locked by a Clopper–Pearson bound; the rule provably controls the population regression rate, needs no training, and applies to repairers that expose KEEP token probabilities. On 11,797 KQA Pro questions, the gate turns a net-negative frozen repairer (net −18) into net +57, with a 95% upper bound of 0.35% on its regression rate against a 0.5% target, and κ achieves higher proposal-ranking AUROC than token likelihood or a prompted judge. Fine-tuned repairers achieve similar net gains (+55 to +57), raising KEEP recall from 71% to 88% without improving error-ranking AUROC. After target-domain recalibration on WebQSP and CWQ, the fine-tuned repairer's gate never exceeds its risk target across 1,000 random splits, but yields no positive net gains.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Large Language Models, Conservative Query Repair, Execution-Realizable Interventions, Structured Prediction, Knowledge Graph Question Answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JmH9OpenReview ↗
Rating
1(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates conservative KBQA query repair as a risk-control problem and proposes a training-free acceptance gate based on the repairer's KEEP-sequence probability. A Clopper–Pearson bound is used to control the population regression rate under specified calibration assumptions. The formulation is useful, but the semantic meaning of the acceptance score and the methodological novelty beyond established risk-control techniques remain concerns.

Critical strengths

The paper explicitly accounts for the asymmetric costs of correcting errors and introducing regressions.

The acceptance rule provides a clearly stated finite-sample guarantee under its assumptions.

Critical weaknesses
  • We appreciate the theoretical contribution (relative repair-gain objective and principled thresholding), but practical importance in the high-accuracy regime remains weakly supported: accuracy changes only from 93.96% to 93.80% (repair) and 94.44% (proposed method).
  • Positive results are mainly for one repair model on KQA Pro; transfer settings show zero or negative net gain, supporting damage limitation rather than accuracy improvement.
  • KEEP depends on model decoding probability, is validated mainly on Qwen3-14B-AWQ, and its advantage over the prompted judge is small; uncertainty and generalizability need clarification.
Questions for authors

Beyond applying an existing risk-control procedure, what is the novel algorithmic contribution?

Reviewer_DJZsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies when a KBQA system should keep its original query and when it should accept a proposed repair. The main concern is that a repair model may fix some wrong answers but also spoil correct ones. The method uses the repair model's probability of returning KEEP on the proposed query, then chooses an acceptance threshold from labeled calibration data. Under the paper's fixed-system and same-distribution assumptions, it bounds the rate of turning correct answers into wrong ones. On KQA Pro, the frozen repairer's net gain changes from -18 to +57. Fine-tuning gives similar gains. Recalibration on WebQSP and CWQ does not produce a positive net gain.

Critical strengths
  1. The problem matters in a system that already answers most questions correctly. Counting both fixes and newly introduced mistakes gives a clear picture of whether repair actually helps. The gate requires no additional model training, although it still needs labeled data to choose its threshold.
  2. The paper explains the assumptions behind its guarantee and separates an overall error-rate bound from the correctness of a single repair. It reports cases where the model is confidently wrong and where transfer gives no net benefit. The main result and its reported counts agree, and the authors do not claim that the gate makes every repair safe.
Critical weaknesses
  1. The guarantee and the official-set result need to be kept clearly separate. Section 3.1 uses an internal BART-based set to select the threshold and a separate deployment cache for official validation. Appendix A.2 correctly notes that these are different cohorts and reports the official-set bound as separate evidence. This limitation should also be easy to see beside the main result. Since both cohorts were used during development, the order in which the prompts, scoring rule, and checking steps were fixed also needs clarification. This is a protocol question, not a claim that the proof is wrong.

  2. The positive result is mainly shown for one repair model on KQA Pro. Table 5 gives zero net gain when all original queries are kept and negative mean net gains in the other transfer settings. The authors acknowledge this and discuss missing retrieved relations as a bottleneck. The result supports limiting damage in those settings, but not improved answer accuracy there. As Section 5 notes, the gate applies existing risk control to the repair setting.

  3. The KEEP score's gain over the prompted judge is small. In Table 2, it gives three more net corrections on the separate reporting half of the development set, while the judge has higher average precision (AP). The authors already note that neither score wins on both ranking measures. The paper supports KEEP probability as a useful score in this experiment, but the size of its advantage needs a clearer uncertainty estimate.

4.The extra group-based loss was tested with a small training budget. Appendix H and Section 3.1 report eight updates per seed, with eight natural examples and one complete group per update. Table 13 shows about 50% accuracy on held-out group examples. This supports the conclusion that the added loss did not help in the tested setup. It does not show what would happen with more training, which the authors also leave open. Appendix H's timing results cover generation logs rather than the full repair-and-scoring process.

Questions for authors
  1. Section 3.1 says both KQA Pro question sets were used during development. Which data were used to choose the prompts, scoring method and query checks? Was the selection half used to make any of these choices, or only to choose the acceptance threshold?

  2. For the Table 5 settings where some repairs are accepted, how many are accepted on average in each test split? What percentage of the test questions is that?

  3. In Table 2, KEEP probability gives 34 net corrections and the prompted judge gives 31. Could you give a confidence interval for this three-question difference, so we can judge how reliable the small gain is?

  4. Why did you use eight training updates? If you have results from longer runs, does the extra group-based loss improve on standard fine-tuning in those runs?

  5. Appendix H gives generation time. What is the total time per question once query execution and KEEP scoring are included? How much extra time does the gate add?

Reviewer_MMSiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies conservative query repair for KBQA parsers. It separates repair proposal generation from repair acceptance, introduces a KEEP-sequence probability k as an acceptance score, and uses a Clopper–Pearson-calibrated threshold to gate proposed repairs with a certified bound on the population regression rate. Experiments on KQA Pro, WebQSP and CWQ, show that certified acceptance can turn a net-negative frozen repairer into a positive net gain while keeping the regression rate below the target.

Critical strengths
  1. The paper presents the proposed method—separating proposal from acceptance and using a k-based certified acceptance gate—and provides a relatively rigorous guarantee showing that the method controls the population regression rate.
  2. The paper validates the method on KQA Pro,WebQSP and CWQ, and the results indicate that CERTIFIED ACCEPTANCE is effective to some extent in reducing wrong edits (regressions).
Critical weaknesses
  1. The motivation is not fully convincing to me. Table 1 shows that with already high accuracy (original 93.96%), applying repair only slightly reduces accuracy (93.80%). Thus the abstract claim that “even a small fraction of wrongly edited correct answers can outweigh gains” does not feel strongly supported in practice; repair itself appears to have little effect here. With the proposed method, the final accuracy is 94.44% and the reported net gain is +57, but from an application perspective the perceived accuracy improvement seems limited. This makes me concerned about the practical value and applicability of repair and of the proposed method, when upstream accuracy is already above 92%. 
  2.  k (line 131) is computed from the KEEP-token decoding probability and is therefore model-dependent. The experiments appear to report results only on Qwen3-14B-AWQ, so it is unclear whether the theoretical claims generalize to other models. Overall, I acknowledge the theoretical contribution of introducing a relative objective for repair gains and a principled thresholding scheme, but I am not yet convinced that the problem is sufficiently important or that the method has strong value in the high-accuracy regime.
Questions for authors

See the weakness sections. They are also questions for authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.