acceptodds
Withdrawn from ICLR 2027

How Much Evidence Should a Self-Correction Carry? Adaptive Dirichlet Evidence for Self-Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-model agents can generate, execute, and revise their own solutions, creating supervision for self-improvement. A correction’s apparent quality, however, need not reveal how concentrated its external support is. We introduce Effective-Evidence Self-Distillation (EESD), which separates relative transition support from an effective pseudo-count mass derived from normalized execution relevance. A Dirichlet posterior converts these quantities into an uncertainty-penalized correction weight for KL-anchored self-distillation. Under a symmetric prior, changing mass preserves category ordering, while effective mass cannot increase the supervised coefficient relative to matched fixed-mass weighting. Across four model–domain history sweeps, increasing visible observations from one to eight reduces EED future-outcome NLL by 55.0–59.3%. At eight observations, EED also achieves lower NLL than fixed mass in all four comparisons. In a matched DeepSeek/RunBugRun study, the predictors agree in argmax on all 3,000 examples, with the largest NLL gain in the most concentrated relevance bin. The complete learning update achieves its largest observed downstream gain on DeepSeek/CodeARC, raising all-tests Pass@1 from 15.0% to 20.4%. These results connect an explicit evidence decomposition to probability estimation and correction learning.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
recursive self-improvement, self-distillation, Uncertainty

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JToXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how much LLM should learn from its self-generated corrections based on execution feedback on code generation tasks. The authors propose Effective-Evidence Self-Distillation (EESD). It separates the relative support for different execution outcomes from the strength of the supporting evidence. It uses a Dirichlet posterior to estimate correction utility and uncertainty, which are then used to weight the training sample/loss. The authors evaluate the method on multiple models and coding tasks and show improvements in outcome prediction and downstream code generation.

Critical strengths
  • The paper motivates a important problem in self-improvement: how much a model should learn from its own corrections. Separating relative execution support from evidence strength is an intuitive idea.

  • The paper provides theoretical guarantees. It includes preservation of category ordering and an upper bound on the supervised weight relative to fixed-mass weighting.

  • The experiments cover multiple models and coding domains, and examine both outcome prediction and downstream code generation.

Critical weaknesses
  • RQ3 experiments compare EESD update with no update, but do not include matched correction SFT, fixed-mass training, or ablations of the uncertainty penalty. Thus, it is unclear whether the observed gains come from their method or simply from training on the corrected programs.

  • Results are a bit mixed. For RQ3, Table 7 reports four positive, one unchanged, and seven negative changes in Pass@1. In Table 10, the NLL benefit of effective mass also changes across different initialization seeds.

  • In Sections 3.1–3.2, the authors use execution relevance scores to compute transition support and effective evidence mass. However, the paper provides little evidence the resulting mass can identify corrections that are more reliable or useful for training. appendix B.3 explains the relevance scores are computed using a frozen lexical similarity between the code diff and execution descriptions, but the effectiveness of this relevance measure is not evaluated. Moreover, much of the improvement from additional execution history in RQ1 is also achieved by the fixed-mass baseline (Table 8). Although lower NLL indicates better outcome prediction, it does not directly validate the proposed uncertainty-based training weights or demonstrate their benefit for correction learning.

Questions for authors
  • Could the authors specify the source-level split and any overlap between correction collection, hyperparameter selection, and downstream evaluation (Appendix B.1)? For RQ1 and RQ2, please also clarify what “future outcome” refers to and how the predicted outcomes are separated from the visible executions.

  • I am confused about how the revised program (a_1) is generated and selected in Section 3.1. Are revisions that still fail some public tests included in training? If so, why is the entire revised program used as the SFT target despite its remaining errors? This matters because the utility vector assigns the same value to PRESERVE and UNRESOLVED. eg, with four uniformly weighted executions, "one FIX and three UNRESOLVED" and "one FIX and three PRESERVE" receive the same training weight, although the revised programs pass one and four tests, respectively.

  • For the execution relevance scores, could the authors provide more details about the text features and hashing procedure used by the 256-dimensional encoder in Appendix B.3? does each execution descriptor contain the results before correction, after correction, or both?

Reviewer_GRLzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Effective-Evidence Self-Distillation (EESD), a method that uses public execution feedback to decide how much a model should learn from program corrections. It separates relevance-weighted transition proportions from the strength assigned to that evidence, measured by the pseudo-count mass (E(p)=1/\sum_i p_i^2). A Dirichlet model combines these quantities into a utility score that penalizes uncertainty. This score weights the cross-entropy loss on corrections, alongside a fixed KL anchor.

The paper establishes ordering invariance and monotonicity properties of the supervised coefficient. It then evaluates outcome prediction and one-round adaptation across three models and four coding domains. The largest downstream improvement is for DeepSeek on CodeARC, where Pass@1 increases from 15.0% to 20.4%.

Critical strengths

The weighting mechanism is explicit and supported by useful analysis: Eqs (3)–(8) clearly distinguish transition support, pseudo-count mass, and correction utility. Proposition 1 establishes that changing the mass preserves category ordering and that using effective mass cannot increase the supervised coefficient relative to a matched fixed-mass alternative. Appendix A.4 identifies when reducing the mass improves or worsens log loss. These results makes it possible to examine what the weighting rule does and where its benefits may depend on the setting and Appendix A.5 appropriately acknowledges that the method does not correct for duplicated or dependent tests.

Critical weaknesses
  • The experiments do not establish that adaptive evidence weighting improves correction learning: Appendix H explicitly confirms that fixed-mass training and equal-weight correction training were not included as controls. Comparing the complete update with no update cannot separate the effect of the proposed evidence weighting from the effects of learning on corrections or KL regularization.

The full downstream results also give reason for caution. Table 7 reports 4 positive, 7 negative, and one unchanged point estimate. Gemma drops from 36/500 to 2/500 on APPS Replay and from 31/500 to 4/500 on CodeContests Replay, with both reported intervals for the change lying below zero. The probability-level intervention does not resolve the attribution problem: Table 10 reports signed NLL gains of (+0.00609), (-0.00367), and (+0.00067) across repetitions. Reporting these results openly is valuable, but they do not establish an advantage for the proposed learning rule.

  • The evaluation protocol does not provide enough detail to reconstruct the experiments or interpret their results confidently: Appendix B.1 explicitly leaves the relationship between the training sources and evaluator sources unestablished. Appendices B.3 and E distinguish the correction scorer from the settings used in the prediction study, but do not fully define the future-outcome label space, how observations are constructed, or the matched “effective-arm” configuration. Table 5 describes response-token accounting without reporting a numerical training budget.

These omissions leave two questions unresolved: I) whether fresh generation represents generalization to unseen tasks, and II) how the prediction results support the mechanism used in training. The problem is insufficient information; the submission does not establish that leakage occurred.

Questions for authors
  1. With the correction records, KL anchor, and training budget held fixed, how does EESD compare with fixed-mass weighting and equal-weight correction training? These comparisons would help establish whether the proposed evidence construction contributes to downstream performance.

  2. Could the authors provide the missing protocol details, including overlap between training and evaluation sources, prediction labels, construction of public and future observations, predictor hyperparameters, and what changes across the initializations in Table 10? These details are needed to interpret and reproduce the evaluation.

  3. How was the relevance-kernel temperature selected, and how sensitive is the correction weighting to that choice? Appendix B.3 uses a frozen signed-hash encoder with kappa=16. Because this temperature affects both relative support and effective evidence mass, could the authors characterize how reasonable changes affect E(p), the fraction of corrections receiving zero weight, and downstream performance? This would clarify whether the weighting behaves robustly or depends strongly on the particular relevance configuration.

Reviewer_kZZROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposed a technique on LLM finetuning on coding by assigning each training example weight based on the tests, where each test has a relevance score using lexical similarity between the code modification and the test description.

Critical strengths

n/a

Critical weaknesses

Honestly I find this paper impossible to read, regardless if it is AI-generated or human-written. Therefore, it is very hard for me to judge the merit of the paper and provide a thorough review. Just to quote a few:

In the abstract: "... separates relative transition support from an effective pseudo-count mass derived from normalized execution relevance. A Dirichlet posterior converts these quantities into an uncertainty-penalized correction weight for KL-anchored self-distillation."

"The key is to separaterelative transition support from total evidence mass..."

And the paper's core idea, if I understand correctly, is just reweight training example with a heuristic that only applies for coding. It is packaged under convoluted equations, and I am not sure if the main assumption that correction should receive less training weight when most of the relevance is concentrated on only a few tests makes sense.

Also I am not sure if this paper has anything to do with self-improvement.

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.