- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Soft Teacher Forcing mixes ground-truth token embeddings with embeddings of the model’s predictive distributions. These distributions come from a detached, teacher-forced pass. A second pass optimizes the usual next-token loss, while inference remains unchanged. Experiments cover four mathematics datasets and seven pretrained models, with five seeds per setting. The method improves GSM8K accuracy by 2.1–4.1 points across models. Ablations examine the mixing distribution, soft-token construction, and alternative regularizers. The measured training overhead is 42% in time and 14% in peak memory
Critical strengths
The gains across seven models are the strongest evidence for the method. On Gemma2-2B, the top-5 variant improves GSM8K accuracy from 44.70% to 49.28%, without changing inference. I put more weight on these SFT improvements than on comparisons with latent-reasoning methods. The authors also test several plausible alternatives, including argmax-based mixing, Gaussian noise, and Bernoulli replacement. Reporting five-seed results and measured training costs makes the practical trade-off easier to assess
Critical weaknesses
The model-selection procedure needs clarification. Appendix A selects the training budget with the highest mean accuracy over five seeds, but does not identify the selection split. If those same evaluation results are then reported as final performance, both the gains and the paired significance tests could be optimistic. Running multiple seeds does not resolve this issue.
The distinction from scheduled sampling also needs cleaner evidence. Prior work already uses two passes and probability-weighted embeddings, so random interpolation is the main methodological change. The paper includes a mean-matched Bernoulli control, but its sampling granularity is unclear on MATH, where Beta is sampled per sequence. The scheduled baseline also uses less soft-input weight on average. These differences make it difficult to attribute the advantage specifically to interpolation.
Questions for authors
-
Which split was used to select training budgets and other hyperparameters? Were the reported test sets excluded from all selection decisions?
-
In Table 4b, does Bernoulli use one draw per sequence on MATH, matching Beta? Please clarify whether sampling granularity and training budgets were controlled in this comparison.
-
Appendix B’s schedule averages roughly 0.25 soft-input weight, whereas Beta averages 0.5. Does Soft-TF still outperform scheduled replacement when average soft-input weight and sampling granularity are matched?
2 comments
Response to Reviewer 5tLG (2/2)
Table 4b Bernoulli results
Thank you. We used the Bernoulli law with the same sampling granularity and training budgets to confirm that Soft-TF with Beta law is still the best. To match Beta and isolate the possibility of the improvement coming from different sampling granularity, we rerun Bernoulli by using one draw per sequence for MATH. Using the full probability vectors gives 13.76±1.11 and using the top-5 values gives 14.12±1.36. The difference holds, so the effect is not only explained by the granularity of alpha sampling. We will add these results. For clarity, we reproduce our tables with these results below:
| Soft tokens | Weight distribution | GSM8K | MATH |
|---|---|---|---|
| – | TF (CoT) | 44.70 ± 0.89 | 14.44 ± 0.91 |
| Full | Beta | 48.52 ± 1.24 | 15.48 ± 1.50 |
| Full | Uniform | 47.61 ± 1.04 | 15.12 ± 0.94 |
| Full | Bernoulli | 47.92 ± 1.01 | 13.76 ± 1.11 |
| Top-5 | Beta | 49.28 ± 0.59 | 16.20 ± 1.33 |
| Top-5 | Uniform | 47.52 ± 0.66 | 13.60 ± 2.02 |
| Top-5 | Bernoulli | 48.63 ± 0.60 | 14.12 ± 1.36 |
Response to Reviewer 5tLG (1/2)
Model selection
In our work, we wanted to have the strongest results for both our method and baselines. We felt that this would be the fairest, since different methods can require different hyperparameters. To this end, we constructed a small validation set* from the training sets of each dataset. We then chose hyperparameters by greedily ablating each one whilst keeping others constant, to avoid combinatorial explosion of hyperparameters. We show examples of these sweeps below for GSM8K and MATH, where the accuracy is that on the corresponding validation sets.
Training budget
Top-5 vs. full distribution
Per-token vs. per-sequence
Using the best hyperparameters from the validation set above, we trained on the full training set and evaluated on the test set. This led to the final results, which we include below again for clarity.
We will clarify this in the revision.
Does one strategy work across all datasets?
Continuing from the point above, we show that we can also simply use the same hyperparameters across datasets, and still observe benefits of Soft-TF.
For instance, we show below results with alpha sampled per token for all datasets, and full probability vectors used to construct the soft tokens.
We also show results with alpha sampled per sequence for all datasets, and top-5 probabilities used to construct the soft tokens:
Distinction from scheduled sampling and more evidence using matched soft-input weight on average.
Thank you. We confirm that Scheduled sampling with matched soft-input weight on average and matched sampling granularity still lags behind Soft-TF.
To isolate the effect of interpolation and to show that despite using a schedule, scheduled sampling stays behind Soft-TF, we rerun scheduled sampling experiments with matched soft-input weight, by setting the probability of teacher forcing to 1 at the first iteration and linearly decaying it to 0, as proposed by the reviewer. We also match the sampling granularity (alpha per token for GSM8k and AQua, per sequence for MATH and MathQA). The results are shown below: Soft-TF still significantly outperforms scheduled sampling.
Results:
*The held-out validation sets consist of 473/7,473 samples from the GSM8K training set, 500/ 12,000 from MATH, 837/29,837 from MathQA, and 1,467/97,467 from AQuA. We chose these sizes such that we keep round numbers for the training sets.