acceptodds
Under review as a conference paper at ICLR 2027

Entropy Regularization: A Free Correction to Cross-Entropy for Verified Demonstrations

Abstract

Large language models are often post-trained on expert demonstrations using cross-entropy (CE), even when the downstream objective is not to imitate the demonstrated solution but to produce any output accepted by a verifier. This mismatch is seen in verifiable domains with multiple correct solutions, such as mathematical reasoning and code generation, where training data may contain only one expert solution per problem. We show that minimizing cross-entropy can be misaligned with minimizing verifier risk; two policies can assign identical likelihood to the observed demonstrations while placing different probability mass on incorrect outputs. This is formalized through a learning-theoretic counterexample in which CE minimization selects a suboptimal policy. We identify that controlling the support of the learned policy can solve this problem by preventing probability mass from spreading to unsupported outputs. Since support size is non-differentiable and computationally intractable, we propose entropy-regularized cross-entropy (ER-CE), using token-level Shannon entropy as a tractable proxy. Finally, across mathematical reasoning and code-generation benchmarks, we find that entropy-regularized training consistently improves verifier accuracy over standard cross-entropy. Our results identify a simple failure mode of imitation-based post-training in verifiable tasks and provide a practical objective that is better aligned with producing correct outputs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Post-training, Statistical Learning Theory, Cross-Entropy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4dNaOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to improve verifier accuracy in supervised fine-tuning from expert demonstrations when a verifier accepts multiple correct answers. The authors found a case in which the widely used cross-entropy (CE) minimization method can fail to PAC-learn verifier risk. In this case, a policy with smaller support avoids the failure, motivating a penalty on the support size of the response distribution. They show that CE with a differentiable approximation of this penalty achieves PAC learnability under certain conditions. In their experiments, they use token-level Shannon entropy as a practical surrogate and empirically demonstrate higher verifier accuracy than standard CE fine-tuning on mathematical reasoning and code-generation tasks.

Critical strengths
  1. The paper provides a theoretical justification for penalizing the size of response-level support. Under certain conditions, minimizing CE with a regularization term that approximates the support size can PAC-learn verifier risk. The counterexample motivates this approach by showing that CE minimization can favor a hypothesis that assigns a higher likelihood to observed demonstrations even though it also assigns probability to incorrect responses.
  2. Motivated by the theory, simply adding token-level Shannon entropy regularization to the fine-tuning objective improves verifier accuracy on mathematical reasoning and code-generation benchmarks.
Critical weaknesses
  1. The experiments do not assess whether minimizing the Shannon entropy of next-token distributions conditioned on expert-demonstration prefixes also reduces the response-level support-size surrogate used in Theorem 4.5.
Questions for authors
  1. Could you evaluate whether token-level Shannon entropy regularization also reduces entropy over complete model-generated responses? This would help clarify how the practical objective relates to the response-level regularization analyzed in Theorem 4.5.
3 comments
AuthorsOpenReview ↗

We thank the reviewer for the positive assessment and for a precise question.

Q1

Yes. We directly measured entropy over complete model-generated responses and found that token-level entropy regularization substantially reduces it.

For 200 GSM8K test problems, we sampled 16 complete responses per problem from the untempered policy (T=1, full softmax, no top-k, top-p, or repetition penalty), using three seeds per setting. We estimated response-level Shannon entropy as the mean negative log-probability of the sampled complete responses.

λResponse-level entropy (nats)Distinct final answers (of 16)Fraction correct
0 (CE)43.57.790.439
117.06.040.528
311.85.360.558
88.54.620.589

As λ increases from 0 to 8, entropy over complete generated responses decreases approximately fivefold (43.5 → 8.5 nats). The number of distinct final answers observed among 16 samples also decreases by 41%, while the fraction of correct responses increases from 43.9% to 58.9%.

These results provide direct empirical evidence connecting our practical token-level regularizer to the response-level entropy considered in our theoretical motivation. Although the penalty is computed only along demonstration prefixes during training, the resulting model exhibits substantially lower entropy over complete self-generated responses. Moreover, this reduction coincides with improved verifier accuracy.

Our response to Reviewer gq87's W3 provides further analysis of the relationship between response-level entropy and verifier risk.

Theory

As the reviewer correctly notes, Theorem 4.5 does not directly cover the objective used in our experiments, which differs from the theoretical objective in three ways: (i) token-level rather than sequence-level entropy regularization, (ii) a fixed regularization coefficient λ\lambda, and (iii) Renyi order α=1\alpha=1. We agree that this distinction should be made clearer. Theorem 4.5 motivates the experimental objective, rather than providing a formal generalization guarantee for it. Nevertheless, the theoretical analysis provides useful insights into each of these choices.

Fixed λ\lambda. The proof of Theorem 4.5 shows that fixing λ>0\lambda>0 introduces an additive error of at most log⁡(1/η)/λ\log(1/\eta)/\lambda in the verifier-risk bound. Thus, this error can be made arbitrarily small by choosing a sufficiently large, but fixed, regularization coefficient.

Renyi order α=1\alpha=1. Under Assumption 4.3, the gap between support size and sequence-level Renyi entropy remains bounded at α=1\alpha=1: 0≤∣supp⁡(p)∣−S1(p)≤1η−1.0\leq |\operatorname{supp}(p)|-S_1(p) \leq \frac{1}{\eta}-1. Consequently, the proof of Theorem 4.5 can be adapted to obtain an approximate generalization guarantee for the sequence-level objective with fixed λ\lambda and α=1\alpha=1. Unlike the limit α→0\alpha\to0, however, this introduces an approximation error that need not vanish. Empirically, our experiments in Appendix G show that Renyi orders α∈[0.5,2]\alpha\in[0.5,2] yield similar improvements over ordinary cross-entropy.

Token-level entropy. The transition from sequence-level to token-level entropy is not formally covered by Theorem 4.5. However, the entropy chain rule provides a theoretical connection. Writing

H^h(x,y)=∑t=1TH(h(⋅∣x,y1:t−1))\hat{H}_h(x,y)=\sum_{t=1}^{T} H(h(\cdot\mid x,y_{1:t-1}))

we have

H(h(⋅∣x))=EY∼h(⋅∣x)[H^h(x,Y)].H(h(\cdot\mid x))=\mathbb{E}_{Y\sim h(\cdot\mid x)}[\hat H_h(x,Y)].

The experimental objective instead evaluates H^h(x,Y)\hat{H}_h(x,Y) along responses sampled from the demonstrator. A change-of-measure argument allows us to relate these quantities. In particular, for a finite vocabulary of size KK,

∣H(h(⋅∣x))−EY∼h∗(⋅∣x)[H^h(x,Y)]∣≤Tlog⁡K12DKL(h∗(⋅∣x)∥h(⋅∣x))\lvert H(h(\cdot\mid x))-\mathbb{E}_{Y\sim h^{\ast}(\cdot\mid x)}[\hat{H}_h(x,Y)]\rvert \leq T\log K\sqrt{\frac{1}{2}D_{\mathrm{KL}}(h^{\ast}(\cdot\mid x)\parallel h(\cdot\mid x))}

Thus, the expected token-level entropy penalty approximates sequence-level Shannon entropy when the learned policy is sufficiently close to the demonstrator in KL divergence. Since the cross-entropy component of our experimental objective also penalizes this divergence at the population level, the two components jointly motivate the use of token-level entropy regularization.

Importantly, this connection does not establish that the experimental objective inherits the PAC-learnability guarantee of Theorem 4.5. Establishing such a result would require further assumptions controlling the difference between token-level entropy and the sequence-level support penalty. Characterizing these conditions is beyond the scope of this work. Nevertheless, our empirical results in Section 5 demonstrate that the proposed objective consistently improves verifier accuracy, providing practical support for this theoretically motivated surrogate.

Thank you for the response. The additional experiments answer my question (Q1) by showing that token-level Shannon entropy regularization also reduces entropy over complete model-generated responses. Further, the authors also clarified that Theorem 4.5 does not directly give a PAC-learnability guarantee for the proposed objective. In light of this clarification, I maintain my positive rating.

Thank you for your positive perspective on our work. We appreciate the reply and the broader review. It prompted us to provide key clarifications.

Reviewer_gq87OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the potential inconsistency between the traditional CE training objective and the final Verifier Risk in tasks with multiple correct solutions. The author first uses a theoretical counterexample to demonstrate that minimizing CE does not necessarily reduce verifier risk. Subsequently, by focusing on controlling the support size of the output distribution, they propose the ER-CE method and establish corresponding PAC learning guarantees. The experiment validates the approach on three benchmarks, the results show that ER-CE improves the model’s greedy Pass@1 accuracy compared to CE. However, the relationship between the theoretical results and the practical training objective, as well as the specific mechanisms underlying the performance improvements, remain to be further clarified.

Critical strengths

(1) This paper focuses on tasks with multiple correct solutions in the fields of mathematical reasoning and code generation. To address potential inconsistencies between the traditional cross-entropy training objective and validation accuracy, the authors propose an improved method based on entropy regularization.

(2) The paper was validated on three benchmarks, and ER-CE achieved higher accuracy than standard CE. Additionally, the authors conducted ablation experiments varying regularization strength, sampling temperature, and entropy order to further analyze the method’s performance, providing empirical evidence of its practical applicability.

Critical weaknesses

Major concerns

(1) The content of the theoretical analysis does not align with the actual training objectives of the subsequent experiment. The entropy penalty is applied at the token level along the prefix of the expert demonstration, whereas the theory considers the distribution of the complete response. Therefore, the theoretical results cannot directly guarantee the effectiveness of the actual training.

(2) The author mentioned that you used the sanitized split of the MBPP dataset, but we found that the official version published on HuggingFace contains only 427 problems, which is inconsistent with your approach of using 300 problems for training and 200 for evaluation. Please clarify how you split the dataset, are the training and evaluation datasets independent?

(3) While the paper does demonstrate that entropy regularization can improve accuracy, it does not fully establish the proposed mechanism, because reducing entropy does not necessarily imply a reduction in the probability mass assigned to incorrect answers. Therefore, the authors need to conduct further experiments to analyze the relationship between changes in entropy and the probability of incorrect answers.

Minor concerns

(1) The experimental results show that the choice of regularization coefficient is very important. However, the paper only lists the results for different parameters without explaining how to select the optimal parameters. We recommend further clarifying the process for hyperparameter selection.

(2) We recommend providing appropriate visualizations of the method’s framework to further illustrate the overall training process and how entropy regularization affects the output probability distribution, thereby helping readers understand the method’s design and mechanism of action.

Questions for authors

(1) Some configurations with lower α values have shown a significant drop in performance. Could the authors further analyze the reasons for this degradation?

(2) Do the advantages observed in the results depend on the LoRA fine-tuning settings? Would similar improvements still be observed in the case of full-parameter fine-tuning?

(3) How did the authors rule out the influence of factors such as inference difficulty and output length on the experimental results?

2 comments
AuthorsOpenReview ↗

We thank the reviewer for the thorough review and for the concrete suggestions. We hope that we've done enough in this rebuttal to alleviate the major concerns.

W1

Please see our response to reviewer 4dNa, Q1.

The measurements under W3 also show empirical changes in the response-level distribution.

W2

The reviewer is right that the numbers are impossible; this was a documentation error. Our code requests 300 training problems from the official sanitized train split, which contains only 120. The request is capped at the split size, so all MBPP experiments used the full 120-problem train split. Evaluation uses 200 fixed-seed samples from the separate 257-problem sanitized test split. The official splits are disjoint, with no overlap, as verified using task IDs.

W3

We directly measured whether entropy regularization reduces probability mass on incorrect answers. Both response-level entropy and verifier risk decrease under our loss.

For 200 GSM8K test problems, we sampled 16 complete responses per problem from the untempered policy (T=1, full softmax, no repetition penalty), using three seeds per setting. As λ\lambda increases from 0 → 1 → 3 → 8:

  • Response-level entropy (mean negative log-probability of sampled responses) decreases from 43.5 → 17.0 → 11.8 → 8.5 nats.
  • Fraction of correct responses increases from 43.9% → 52.8% → 55.8% → 58.9%, reducing verifier risk from 56.1% to 41.1%.
  • Distinct final answers among 16 samples decrease from 7.79 → 6.04 → 5.36 → 4.62, a 41% reduction in diversity.

Crucially, lower entropy accompanies greater concentration on correct answers. Problems answered correctly in all 16 samples rise from 4.0% to 23.8%, and those answered correctly by a majority rise from 40.3% to 59.0%. Meanwhile, problems never answered correctly change only slightly (12.2% to 9.8%).

These results suggest the penalty primarily concentrates mass on correct solutions for problems the model can sometimes solve, rather than simply collapsing outputs or unlocking previously unsolvable problems.

These are complete-response-level measurements, not token-level. They directly show that our token-level regularizer is associated with reduced response-level entropy and, importantly, reduced mass on verifier-rejected outputs.

Appendix I provides complementary evidence: removing chain-of-thought and training on answer-only targets largely eliminates the benefits or makes them negative at larger λ\lambda, consistent with our mechanism.

W4

Here, we select λ\lambda using held-out validation. On GSM8K, validation selects λ=6\lambda=6, yielding +9.64 test points over CE (95% CI [+7.35, +11.93], p=0.0003). Gains are robust across strengths: after Holm correction, every λ≥2\lambda\geq 2 significantly improves over CE.

Our original experiments aimed to show robustness across strengths rather than select an optimal λ\lambda. Validation selection confirms substantial gains without test-set hyperparameter tuning. See our response to reviewer 2N8K, Q3, for the full selection procedure and statistical analysis.

W5

We agree and will add this, using W3's measurements to trace the connection from our loss to reduced verifier risk.

Q1

Several possible explanations are worth mentioning. Penalty strength is not responsible: at matched final training entropy, α≤0.25\alpha\leq 0.25 underperforms α≥0.5\alpha\geq 0.5 by 8–12 points, and α=0.25\alpha=0.25 at λ=8\lambda=8 is 38 points worse. Nor is numerical error: at α=0.25\alpha=0.25, only 1.6–5.3% of ∑p(y)α\sum p(y)^\alpha comes from tokens outside the top 100.

We hypothesize that for α<1\alpha<1, gradients spread across low-probability tokens, allowing suppression of an irrelevant tail without improving top-token correctness. At α=0.1\alpha=0.1, large penalty gradients (norm 7–10 versus clipping threshold 1.0) also attenuate the CE signal. Thus, α=1\alpha=1 may lie in the effective range rather than being a tractability-driven compromise.

Q2

No. Under full-parameter fine-tuning (learning rate 1e-5, three seeds), every λ\lambda improves over CE: +4.07, +4.40, and +5.40 points at λ=1,3,8\lambda=1,3,8. At validation-selected λ=8\lambda=8, the gain is +5.40 (paired 95% CI [+1.65, +9.15], p=0.025; problem-level bootstrap [+2.53, +8.60]). With three seeds, Holm-adjusted p-values across the three λ\lambda are 0.053–0.057.

Q3

Difficulty

Splitting GSM8K test problems into terciles by reference-solution reasoning steps, validation-selected λ\lambda improves every tercile: +8.3 points (easy), +4.7 (middle), and +13.5 (hard). Gains are largest on hard problems, not concentrated on easy ones; the same pattern holds under full fine-tuning (+7.0, +2.6, +6.2).

Length

Mean generation length changes little: GSM8K, 84 tokens under CE versus 80–81 at λ≥4\lambda\geq 4; MATH, 224 versus 218–221 at λ∈{1,2,4}\lambda\in\{1,2,4\} and 203 at λ=8\lambda=8.

AuthorsOpenReview ↗

Less than 48 hours until the discussion window closes.

Thank you again for your review.

We address your comments and questions in our answers above.

We hope we've clarified all of your concerns and more firmly established the mechanism by which our method works and the relevant experimental details.

If you have further questions, we are very happy to answer them.

Reviewer_2N8KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies supervised fine-tuning with one verified demonstration per problem. It argues that cross-entropy training may assign too much probability to incorrect responses and proposes ER-CE, which adds an entropy penalty to the standard SFT objective.

The paper provides theoretical results about CE's limitations and shows that ER-CE improves over the CE baseline on GSM8K, MBPP, and MATH.

The results are promising, but the theoretical guarantees do not directly apply to the proposed training objective, and the experiments leave several important questions unanswered.

Critical strengths
  1. The paper provides a clear example showing that empirical CE minimization can fail to minimize verifier risk, and proves that a support-based regularizer can address this problem under certain assumptions.

  2. ER-CE outperforms the reported CE baseline across all three benchmarks, with improvements observed across multiple random seeds.

  3. The paper provides per-seed results and additional experiments with sampling and different entropy orders, including some negative findings.

Critical weaknesses
  1. Theorem 4.5 analyzes a response-level Rényi entropy penalty with λm→∞\lambda_m \to \infty and αm→0\alpha_m \to 0, while ER-CE uses fixed-weight, token-level Shannon entropy (α=1\alpha=1). These are different objectives, and no theoretical result covers the actual ER-CE implementation. Although Appendix H acknowledges this gap, the main text still claims that the theory justifies the proposed objective. The authors should clarify this distinction or provide additional theoretical support.

  2. ER-CE is compared mainly against a single CE training configuration, without evaluating the original checkpoints or tuning CE's learning rate and training epochs. There is also no comparison with existing entropy-based methods such as EM-FT. Moreover, the improvement under sampled decoding may partly come from reducing the effective temperature rather than suppressing incorrect responses, as Appendix G.4 acknowledges. Evaluating the starting checkpoints, a tuned CE baseline, EM-FT, and temperature-adjusted decoding would provide stronger evidence.

  3. The best regularization strength λ\lambda is selected using the evaluation results, without a separate validation split, which may overestimate the gains. The paper also claims statistical significance but reports only means and standard deviations across 3–5 seeds, without appropriate statistical tests. Finally, the MBPP experiment reports 300 training and 200 evaluation problems from a dataset containing only 426 problems. Unless a different sampling procedure was used, these sets cannot be disjoint, raising concerns about possible data leakage. The authors should clarify the split and report results with proper validation and statistical testing.

Questions for authors
  1. What theoretical guarantee applies to the actual ER-CE objective using fixed-weight, token-level Shannon entropy?

  2. How does ER-CE compare with the original pretrained checkpoints, a tuned CE baseline, and existing entropy regularization methods?

  3. Can the authors select the regularization strength on a separate validation set and report confidence intervals or statistical tests?

  4. Can the authors clarify the MBPP data split and confirm that training and evaluation examples do not overlap?

  5. Does ER-CE still improve sampled accuracy when the CE baseline uses a tuned decoding temperature?

2 comments
AuthorsOpenReview ↗

We thank the reviewer for a careful and specific review. This review motivated theoretical and empirical insights that we hope will strengthen the message of our work.

Q1.

We ask the reviewer to see our response to reviewer 4dNa, Q1.

Q2.

Base Checkpoints

Qwen 1.5B Instruct scores 0.458 on GSM8K, versus 0.488 after CE and 0.585 after ER-CE; Qwen 7B Instruct scores 0.415 on MATH, versus 0.420 and 0.472. On MBPP, the base scores 0.555; CE lowers it to 0.527, while ER-CE peaks at 0.571.

Since fine-tuning gains are small on instruct checkpoints, we repeated GSM8K with base Qwen2.5-1.5B, where fine-tuning is more impactful (0.176 before, 0.562 after CE). Here ER-CE gains +2.64 points at validation-selected λ=8\lambda=8, positive over five seeds (paired 95% CI [−1.82, +7.10]; problem-level bootstrap [−0.12, +5.36]).

CE Tuning

ER-CE retains a significant +6.27-point improvement after tuning CE. We swept CE learning rates (5e-5, 1e-4, 2e-4) and epochs (1–3), with three seeds per configuration and held-out validation selection. Validation selected 5e-5 for one epoch, yielding 49.3% test accuracy.

Adding entropy regularization improves accuracy by +6.27 points at validation-selected λ=1\lambda=1 (95% CI [+5.23, +7.30], p=0.002), and +9.40 points at λ=8\lambda=8. Even the best of nine CE configurations achieves only 52.1%, below every ER-CE configuration with λ≥2\lambda\geq 2 under our original settings (54.6–58.5%).

The improvement is not an artifact of insufficient CE tuning.

EM-FT comparison

EM-FT can outperform ER-CE, but requires substantially more compute and careful learning-rate tuning.

We evaluated EM-FT on GSM8K using the same learning-rate and epoch sweep as CE (5e-5, 1e-4, 2e-4; 1–3 epochs), with validation-based selection. EM-FT achieved only 42.60 ± 2.41%, below CE and ER-CE. Its output distributions had extremely low entropy, suggesting collapse.

Expanding the sweep to 1e-6, 5e-6, and 1e-5, validation selected 1e-5, yielding 63.40 ± 0.43%, above ER-CE's 58.16%. Thus, EM-FT achieves higher accuracy but is more learning-rate-sensitive. ER-CE never decreased performance relative to CE at any tested learning rate, whereas EM-FT did.

This comes at approximately 9.6× the training compute. On identical 4×A100 hardware, ER-CE took 1.7 minutes (0.113 GPU-hours) versus EM-FT's 16.3 minutes (1.087 GPU-hours). EM-FT requires sampling model responses during training; ER-CE regularizes demonstrations directly without rollouts.

This reveals an accuracy–compute tradeoff: tuned EM-FT achieves higher accuracy at greater cost, while ER-CE provides a simple, rollout-free correction to CE, consistent with our emphasis on a free correction.

Q3

ER-CE retains a significant +9.64-point improvement over CE with λ\lambda selected exclusively on held-out validation data.

We repeated the GSM8K sweep using 200 validation problems disjoint from the 1,000 training problems. Results reproduce Table 1 within 1.5 points at every λ\lambda. Validation selects λ=6\lambda=6, giving +9.64 points (paired 95% CI [+7.35, +11.93], p=0.0003; bootstrap CI [+6.72, +12.60]). Per-seed validation selection also yields +8.44 points (95% CI [+4.76, +12.12], p=0.003).

Gains are robust across λ\lambda, not dependent on a single favorable choice. After Holm correction, improvements are significant at every λ≥2\lambda\geq 2 on GSM8K; on MBPP and MATH, at λ∈{0.7,1,1.5}\lambda\in\{0.7, 1, 1.5\} and λ∈{1,4}\lambda\in\{1, 4\}, respectively.

Without selecting λ\lambda, averaging improvements across all tested λ>0\lambda>0 gives +5.14 points on GSM8K, +2.30 on MBPP, and +3.92 on MATH.

Thus, gains persist under validation-based selection and are not driven by test-set optimization.

Q4.

Please see our response to reviewer gq87, W2. This was a reporting error; we double-checked that training and test sets were disjoint.

Q5.

Yes. All main-paper results use greedy decoding, invariant to temperature rescaling; hence gains cannot be explained by effective-temperature changes. Appendix G provides the requested results and explains our choice of Greedy Pass@1.

We also repeated temperature-sampling experiments on GSM8K (200 problems, 3 seeds). CE accuracy at T=0.4,0.6,0.8,1.0,1.2T=0.4, 0.6, 0.8, 1.0, 1.2 was 0.471, 0.434, 0.383, 0.295, 0.164. Accuracy decreases monotonically with TT and approaches greedy accuracy (0.472) as T→0T\to 0. Thus, CE reaches at most approximately 0.472 across tested temperatures, while ER-CE reaches 0.573 at T=0.8T=0.8.

AuthorsOpenReview ↗

Less than 48 hours until the discussion window closes.

Thank you again for your review.

We address your comments and questions in our answers above.

We hope we've clarified the role of our theory in motivating the tractable objective and the guarantees that are conferred by the objective we've used. We also hope to have provided relevant ablations and clarifications about our experiments and empirical results.

If you have further questions, we are very happy to answer them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviews and Rebuttals for AC

Dear Area Chair,

We thank the reviewers for their constructive feedback. We summarize how our rebuttal addressed their main concerns.

Reviewer 2N8K (2): We addressed the theoretical concerns by clarifying that our PAC-learning guarantee applies to a response-level Rényi entropy objective under some assumptions, while the practical token-level Shannon entropy objective is motivated by this theory but does not directly inherit its guarantee. We did show that our tractable objective empirically decreases the entropy on which the PAC guarantee rests. We also substantially strengthened the baselines. ER-CE retains a statistically significant +6.27-point improvement over a tuned CE baseline, and a +9.64-point improvement with validation-selected regularization strength. We further compared against pretrained checkpoints and EM-FT, finding that while tuned EM-FT achieves higher accuracy after extensive tuning, it requires approximately 9.6× more training compute. We emphasize that our method is essentially free. Finally, we clarified the MBPP split and showed that the improvements cannot be explained by decoding temperature alone.

Reviewer gq87 (2): We provided new experiments to examine the mechanism behind our method. Response-level entropy decreases substantially under ER-CE, while the fraction of correct sampled responses increases from 43.9% to 58.9%, providing evidence that the regularization shifts probability mass away from incorrect outputs. We also demonstrated that the improvements persist under full-parameter fine-tuning (+5.40 points) and analyzed performance across problem difficulties and output lengths.

Reviewer 4dNa (3): We directly addressed the reviewer's main question by measuring entropy over complete model-generated responses, finding an approximately fivefold reduction under our regularization. We also clarified the theoretical connection and its limitations. The reviewer subsequently confirmed that the additional experiments answered their question and maintained their positive rating.

We thank the AC for their consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.