- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper aims to improve verifier accuracy in supervised fine-tuning from expert demonstrations when a verifier accepts multiple correct answers. The authors found a case in which the widely used cross-entropy (CE) minimization method can fail to PAC-learn verifier risk. In this case, a policy with smaller support avoids the failure, motivating a penalty on the support size of the response distribution. They show that CE with a differentiable approximation of this penalty achieves PAC learnability under certain conditions. In their experiments, they use token-level Shannon entropy as a practical surrogate and empirically demonstrate higher verifier accuracy than standard CE fine-tuning on mathematical reasoning and code-generation tasks.
Critical strengths
- The paper provides a theoretical justification for penalizing the size of response-level support. Under certain conditions, minimizing CE with a regularization term that approximates the support size can PAC-learn verifier risk. The counterexample motivates this approach by showing that CE minimization can favor a hypothesis that assigns a higher likelihood to observed demonstrations even though it also assigns probability to incorrect responses.
- Motivated by the theory, simply adding token-level Shannon entropy regularization to the fine-tuning objective improves verifier accuracy on mathematical reasoning and code-generation benchmarks.
Critical weaknesses
- The experiments do not assess whether minimizing the Shannon entropy of next-token distributions conditioned on expert-demonstration prefixes also reduces the response-level support-size surrogate used in Theorem 4.5.
Questions for authors
- Could you evaluate whether token-level Shannon entropy regularization also reduces entropy over complete model-generated responses? This would help clarify how the practical objective relates to the response-level regularization analyzed in Theorem 4.5.
3 comments
Thank you for the response. The additional experiments answer my question (Q1) by showing that token-level Shannon entropy regularization also reduces entropy over complete model-generated responses. Further, the authors also clarified that Theorem 4.5 does not directly give a PAC-learnability guarantee for the proposed objective. In light of this clarification, I maintain my positive rating.
Thank you for your positive perspective on our work. We appreciate the reply and the broader review. It prompted us to provide key clarifications.
We thank the reviewer for the positive assessment and for a precise question.
Q1
Yes. We directly measured entropy over complete model-generated responses and found that token-level entropy regularization substantially reduces it.
For 200 GSM8K test problems, we sampled 16 complete responses per problem from the untempered policy (T=1, full softmax, no top-k, top-p, or repetition penalty), using three seeds per setting. We estimated response-level Shannon entropy as the mean negative log-probability of the sampled complete responses.
As λ increases from 0 to 8, entropy over complete generated responses decreases approximately fivefold (43.5 → 8.5 nats). The number of distinct final answers observed among 16 samples also decreases by 41%, while the fraction of correct responses increases from 43.9% to 58.9%.
These results provide direct empirical evidence connecting our practical token-level regularizer to the response-level entropy considered in our theoretical motivation. Although the penalty is computed only along demonstration prefixes during training, the resulting model exhibits substantially lower entropy over complete self-generated responses. Moreover, this reduction coincides with improved verifier accuracy.
Our response to Reviewer gq87's W3 provides further analysis of the relationship between response-level entropy and verifier risk.
Theory
As the reviewer correctly notes, Theorem 4.5 does not directly cover the objective used in our experiments, which differs from the theoretical objective in three ways: (i) token-level rather than sequence-level entropy regularization, (ii) a fixed regularization coefficient λ, and (iii) Renyi order α=1. We agree that this distinction should be made clearer. Theorem 4.5 motivates the experimental objective, rather than providing a formal generalization guarantee for it. Nevertheless, the theoretical analysis provides useful insights into each of these choices.
Fixed λ. The proof of Theorem 4.5 shows that fixing λ>0 introduces an additive error of at most log(1/η)/λ in the verifier-risk bound. Thus, this error can be made arbitrarily small by choosing a sufficiently large, but fixed, regularization coefficient.
Renyi order α=1. Under Assumption 4.3, the gap between support size and sequence-level Renyi entropy remains bounded at α=1: 0≤∣supp(p)∣−S1(p)≤η1−1. Consequently, the proof of Theorem 4.5 can be adapted to obtain an approximate generalization guarantee for the sequence-level objective with fixed λ and α=1. Unlike the limit α→0, however, this introduces an approximation error that need not vanish. Empirically, our experiments in Appendix G show that Renyi orders α∈[0.5,2] yield similar improvements over ordinary cross-entropy.
Token-level entropy. The transition from sequence-level to token-level entropy is not formally covered by Theorem 4.5. However, the entropy chain rule provides a theoretical connection. Writing
H^h(x,y)=∑t=1TH(h(⋅∣x,y1:t−1))
we have
H(h(⋅∣x))=EY∼h(⋅∣x)[H^h(x,Y)].
The experimental objective instead evaluates H^h(x,Y) along responses sampled from the demonstrator. A change-of-measure argument allows us to relate these quantities. In particular, for a finite vocabulary of size K,
∣H(h(⋅∣x))−EY∼h∗(⋅∣x)[H^h(x,Y)]∣≤TlogK21DKL(h∗(⋅∣x)∥h(⋅∣x))
Thus, the expected token-level entropy penalty approximates sequence-level Shannon entropy when the learned policy is sufficiently close to the demonstrator in KL divergence. Since the cross-entropy component of our experimental objective also penalizes this divergence at the population level, the two components jointly motivate the use of token-level entropy regularization.
Importantly, this connection does not establish that the experimental objective inherits the PAC-learnability guarantee of Theorem 4.5. Establishing such a result would require further assumptions controlling the difference between token-level entropy and the sequence-level support penalty. Characterizing these conditions is beyond the scope of this work. Nevertheless, our empirical results in Section 5 demonstrate that the proposed objective consistently improves verifier accuracy, providing practical support for this theoretically motivated surrogate.