acceptodds
Under review as a conference paper at ICLR 2027

Soft Teacher Forcing

Abstract

Soft tokens have so far been used to replace discrete chain-of-thought in latent reasoning methods. These approaches typically require multi-stage training and, in some cases, auxiliary modules, yet they still underperform supervised fine-tuning (SFT) on input, chain-of-thought, output triples. In this paper, we ask whether soft tokens can be used to improve SFT itself, instead of as an avenue for latent reasoning. We introduce Soft Teacher Forcing (Soft-TF), a simple training algorithm that combines teacher-forced embeddings with soft tokens through random convex combinations. The soft tokens are computed in a single parallel forward pass over the ground-truth sequence. Soft-TF adds no parameters, no external model and no curriculum, and leaves inference unchanged. Experiments show that Soft-TF improves over SFT on GSM8K, MATH, MathQA and AQuA, by +2.1 to +4.1 points on GSM8K across seven Gemma, Llama and Mistral models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, SFT, soft tokens

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5tLGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Soft Teacher Forcing mixes ground-truth token embeddings with embeddings of the model’s predictive distributions. These distributions come from a detached, teacher-forced pass. A second pass optimizes the usual next-token loss, while inference remains unchanged. Experiments cover four mathematics datasets and seven pretrained models, with five seeds per setting. The method improves GSM8K accuracy by 2.1–4.1 points across models. Ablations examine the mixing distribution, soft-token construction, and alternative regularizers. The measured training overhead is 42% in time and 14% in peak memory

Critical strengths

The gains across seven models are the strongest evidence for the method. On Gemma2-2B, the top-5 variant improves GSM8K accuracy from 44.70% to 49.28%, without changing inference. I put more weight on these SFT improvements than on comparisons with latent-reasoning methods. The authors also test several plausible alternatives, including argmax-based mixing, Gaussian noise, and Bernoulli replacement. Reporting five-seed results and measured training costs makes the practical trade-off easier to assess

Critical weaknesses

The model-selection procedure needs clarification. Appendix A selects the training budget with the highest mean accuracy over five seeds, but does not identify the selection split. If those same evaluation results are then reported as final performance, both the gains and the paired significance tests could be optimistic. Running multiple seeds does not resolve this issue.

The distinction from scheduled sampling also needs cleaner evidence. Prior work already uses two passes and probability-weighted embeddings, so random interpolation is the main methodological change. The paper includes a mean-matched Bernoulli control, but its sampling granularity is unclear on MATH, where Beta is sampled per sequence. The scheduled baseline also uses less soft-input weight on average. These differences make it difficult to attribute the advantage specifically to interpolation.

Questions for authors
  • Which split was used to select training budgets and other hyperparameters? Were the reported test sets excluded from all selection decisions?

  • In Table 4b, does Bernoulli use one draw per sequence on MATH, matching Beta? Please clarify whether sampling granularity and training budgets were controlled in this comparison.

  • Appendix B’s schedule averages roughly 0.25 soft-input weight, whereas Beta averages 0.5. Does Soft-TF still outperform scheduled replacement when average soft-input weight and sampling granularity are matched?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5tLG (1/2)

Model selection

In our work, we wanted to have the strongest results for both our method and baselines. We felt that this would be the fairest, since different methods can require different hyperparameters. To this end, we constructed a small validation set* from the training sets of each dataset. We then chose hyperparameters by greedily ablating each one whilst keeping others constant, to avoid combinatorial explosion of hyperparameters. We show examples of these sweeps below for GSM8K and MATH, where the accuracy is that on the corresponding validation sets.

Training budget

IterationsGSM8K SFTGSM8K SSGSM8K Soft-TFMATH SFTMATH SSMATH Soft-TF
1k39.3235.7335.94–––
5k41.2342.4943.7615.4015.2013.60
10k35.3136.7940.3813.6015.8016.60
15k–––11.6012.6011.80

Top-5 vs. full distribution

SettingGSM8K SSGSM8K Soft-TFMATH SSMATH Soft-TF
Top-542.4943.7615.8016.60
Full41.8642.9215.0016.00

Per-token vs. per-sequence

SettingGSM8K SSGSM8K Soft-TFMATH SSMATH Soft-TF
Per-token43.5547.9913.4013.80
Per-sequence42.4943.7615.8016.60

Using the best hyperparameters from the validation set above, we trained on the full training set and evaluated on the test set. This led to the final results, which we include below again for clarity.

MethodGSM8KMATHMathQAAQuA
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
SS (Mihaylova & Martins, 2019)46.38 ± 1.0215.04 ± 1.7659.78 ± 1.0139.45 ± 2.32
Soft-TF (ours)49.28 ± 0.5916.20 ± 1.3361.49 ± 0.9642.06 ± 1.95

We will clarify this in the revision.

Does one strategy work across all datasets?

Continuing from the point above, we show that we can also simply use the same hyperparameters across datasets, and still observe benefits of Soft-TF.

For instance, we show below results with alpha sampled per token for all datasets, and full probability vectors used to construct the soft tokens.

MethodGSM8KMATHMathQAAQuA
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
Soft-TF48.52 ± 1.2415.00 ± 1.2760.55 ± 0.5342.06 ± 1.95

We also show results with alpha sampled per sequence for all datasets, and top-5 probabilities used to construct the soft tokens:

MethodGSM8KMATHMathQAAQuA
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
Soft-TF48.46 ± 1.1016.20 ± 1.3361.23 ± 0.6439.21 ± 2.24

 

Distinction from scheduled sampling and more evidence using matched soft-input weight on average.

Thank you. We confirm that Scheduled sampling with matched soft-input weight on average and matched sampling granularity still lags behind Soft-TF.

To isolate the effect of interpolation and to show that despite using a schedule, scheduled sampling stays behind Soft-TF, we rerun scheduled sampling experiments with matched soft-input weight, by setting the probability of teacher forcing to 1 at the first iteration and linearly decaying it to 0, as proposed by the reviewer. We also match the sampling granularity (alpha per token for GSM8k and AQua, per sequence for MATH and MathQA). The results are shown below: Soft-TF still significantly outperforms scheduled sampling.

Results:

MethodGSM8KMATHMathQAAQuA
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
SS46.38 ± 1.0215.04 ± 1.7659.78 ± 1.0139.45 ± 2.32
SS-matched46.51 ± 0.8914.12 ± 1.5159.98 ± 0.5239.84 ± 1.56
Soft-TF49.28 ± 0.5916.20 ± 1.3361.49 ± 0.9642.06 ± 1.95

 


*The held-out validation sets consist of 473/7,473 samples from the GSM8K training set, 500/ 12,000 from MATH, 837/29,837 from MathQA, and 1,467/97,467 from AQuA. We chose these sizes such that we keep round numbers for the training sets.

AuthorsOpenReview ↗

Response to Reviewer 5tLG (2/2)

Table 4b Bernoulli results

Thank you. We used the Bernoulli law with the same sampling granularity and training budgets to confirm that Soft-TF with Beta law is still the best. To match Beta and isolate the possibility of the improvement coming from different sampling granularity, we rerun Bernoulli by using one draw per sequence for MATH. Using the full probability vectors gives 13.76±1.11 and using the top-5 values gives 14.12±1.36. The difference holds, so the effect is not only explained by the granularity of alpha sampling. We will add these results. For clarity, we reproduce our tables with these results below:

Soft tokensWeight distributionGSM8KMATH
–TF (CoT)44.70 ± 0.8914.44 ± 0.91
FullBeta48.52 ± 1.2415.48 ± 1.50
FullUniform47.61 ± 1.0415.12 ± 0.94
FullBernoulli47.92 ± 1.0113.76 ± 1.11
Top-5Beta49.28 ± 0.5916.20 ± 1.33
Top-5Uniform47.52 ± 0.6613.60 ± 2.02
Top-5Bernoulli48.63 ± 0.6014.12 ± 1.36
Reviewer_UdHvOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether soft tokens, so far used mainly to replace discrete chain-of-thought (CoT) in latent reasoning, can instead improve standard supervised fine-tuning (SFT). It argues that recursive latent-reasoning methods such as CoCoNut drift away from teacher forcing and are costly to train, and proposes Soft Teacher Forcing (Soft-TF), which stays close to teacher forcing.

At each training step, one extra forward pass without gradients over the ground-truth sequence gives the model's next-token distribution at every response position. Mapping each distribution through the input embedding matrix gives one soft token per position. Each response-token input embedding is then replaced by a random mix of this soft token and the ground-truth embedding, and the usual next-token loss is applied. The mixing weight is drawn from Beta(0.1, 0.1), per token or per sequence depending on the dataset. Soft-TF adds no parameters and keeps standard greedy decoding at inference.

All models are fine-tuned with LoRA and evaluated over 5 seeds. On Gemma2-2B, Soft-TF improves over standard SFT (TF (CoT)) on GSM8K, MATH, MathQA and AQuA, and in mean accuracy it also exceeds label smoothing, scheduled sampling, and CoCoNut and SoftThinking variants. The evaluation is limited to math datasets and LoRA models of at most 9B parameters.

Critical strengths
  1. Soft-TF is a cheap drop-in change to SFT that gives consistent gains over the standard TF (CoT) baseline. It costs one extra forward pass without gradients, with no new parameters and no change at inference, and improves GSM8K on all seven models from three families by +2.10 to +4.08 points.

  2. The ablations show which design choices matter when training with soft tokens. Convex combinations of vocabulary embeddings outperform last-layer hidden states. Sequential soft-token computation degrades once the number of passes K exceeds a few, and every constant alpha falls below random alpha.

Critical weaknesses
  1. The novelty relative to scheduled sampling is limited. Scheduled sampling for transformers [1] already feeds a second decoder pass with embedding mixtures weighted by the predicted distribution, and the stated difference is interpolation with a weight drawn from a fixed Beta(0.1, 0.1) instead of replacement under a decaying schedule.

  2. Model choices appear to be made on the test sets. No validation split is described, and each method's training budget is the one with the highest mean accuracy over 5 seeds. The soft-token variant (top-5 or full), the alpha sampling unit (per token or per sequence) and the CoCoNut-dagger decoding strategy also vary across datasets or tables, and changing the budget alone moves the Gemma3-1B/GSM8K gain between -0.53 and +3.75 points.

[1] T. Mihaylova and A. F. T. Martins. Scheduled sampling for transformers. arXiv:1906.07651, 2019.

Questions for authors
  1. Was a held-out validation split used for any of the choices listed in weakness 2 or for beta? If not, please re-run the Gemma2-2B comparison of Table 1 (at least TF (CoT), scheduled sampling and Soft-TF) with all choices fixed on a validation split taken from the training set.

  2. Sec. 2.3 says alpha "is sampled at each iteration", while App. A samples it per token on GSM8K/AQuA and per sequence on MATH/MathQA. Which granularity do Tables 2-6 and Fig. 2 use, including the Bernoulli rows of Table 4b, and how was the per-dataset choice made? If the Bernoulli rows use per-token sampling, the larger MATH gap between Beta and Bernoulli (16.20 vs. 13.44; Table 4b) may reflect granularity rather than the distribution.

  3. In Table 10, CoCoNut-dagger scores 1.00 on GSM8K and 0.04 on MATH under mixed and soft decoding, and its best MathQA and AQuA scores (12.14 and 12.05) are below the 20% chance level of five-way multiple choice. Mixed and soft decoding also give identical results on all four datasets. Since these baselines support the claim that Soft-TF outperforms latent reasoning methods, please report answer-format failure rates and confirm that gamma is applied.

  4. Could you report the four-dataset comparison for at least one non-Gemma model (e.g., Llama3.2-3B or Mistral-7B-v0.3) and at least one full fine-tuning run? Either would support the general claims more directly.

5 comments
AuthorsOpenReview ↗

Response to Reviewer UdHv (1/3)

Novelty relative to scheduled sampling

We agree that Soft-TF is closely related to Mihaylova & Martins (2019), which we discuss in Sections 3.2 and 4 and compare against in Table 1. The key difference is the motivation which leads to the difference in design choice. Mihaylova & Martins’ motivation is to fix the train-test discrepancy: training with teacher forcing causes exposure bias since sampling is autoregressive, so following scheduled sampling they progressively replace ground-truth tokens with model predictions. In their setting, probability-weighted embeddings are only one among several replacement variants they consider. We start from a nearly opposite observation: latent reasoning methods fail because they deviate too far from teacher forcing (Sections 2.2, 3.2, 3.3). Soft-TF uses soft tokens for their expressivity while keeping training close to teacher forcing, which is why we interpolate randomly rather than using a schedule that progressively departs from teacher forcing. Table 1 empirically shows the advantage of Soft-TF.

To better assess the effect of random interpolation, and following a suggestion from Reviewer 5tLG, we ran a new comparison.

One factor that changes between Soft-TF and scheduled sampling is how much of each method’s input comes from the model’s own predictions rather than ground-truth tokens. In Soft-TF, each input mixes the ground-truth token and the model’s soft prediction whose weight is drawn from a Beta distribution with mean 0.5. In the scheduled sampling baseline, model prediction usage grows over training but averages only 0.25 (See appendix B: the probability of teacher forcing decreases from 1 at the beginning of training to 0.5 at the end). Soft-TF’s advantage could therefore simply come from using more soft tokens. To falsify this, we rerun scheduled sampling so that the probability of using ground-truth tokens decays linearly from 1 to 0, which also gives an average of 0.5. We also make the mixing the same as Soft-TF (per token/per sequence). We call this variant SS-matched. The remaining difference from Soft-TF is how soft tokens are used: random interpolation vs replacement based on a schedule.

Results:

MethodGSM8KMATHMathQAAQuA
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
SS46.38 ± 1.0215.04 ± 1.7659.78 ± 1.0139.45 ± 2.32
SS-matched46.51 ± 0.8914.12 ± 1.5159.98 ± 0.5239.84 ± 1.56
Soft-TF49.28 ± 0.5916.20 ± 1.3361.49 ± 0.9642.06 ± 1.95

Matching the amount of soft tokens does not help scheduled sampling, and Soft-TF still outperforms it on all four datasets. Thus, the gain comes from random interpolation not from larger soft-input weight.

AuthorsOpenReview ↗

Response to Reviewer UdHv (2/3)

Hyperparameter selection

In our work, we wanted to have the strongest results for both our method and baselines. We felt that this would be the fairest, since different methods can require different hyperparameters. To this end, we constructed a small held-out validation set* from the training set of each dataset. We then chose hyperparameters by greedily ablating each one whilst keeping others constant, to avoid combinatorial explosion of hyperparameters. We show examples of these sweeps below for GSM8K and MATH, where the accuracy is that on the corresponding validation sets.

Training budget

IterationsGSM8K SFTGSM8K SSGSM8K Soft-TFMATH SFTMATH SSMATH Soft-TF
1k39.3235.7335.94–––
5k41.2342.4943.7615.4015.2013.60
10k35.3136.7940.3813.6015.8016.60
15k–––11.6012.6011.80

Top-5 vs. full distribution

SettingGSM8K SSGSM8K Soft-TFMATH SSMATH Soft-TF
Top-542.4943.7615.8016.60
Full41.8642.9215.0016.00

Per-token vs. per-sequence

SettingGSM8K SSGSM8K Soft-TFMATH SSMATH Soft-TF
Per-token43.5547.9913.4013.80
Per-sequence41.8642.9215.8016.60

Using the best hyperparameters from the validation set above, we trained on the full training set and evaluated on the test set. This led to the final results, which we include below again for clarity.

MethodGSM8KMATHMathQAAQuA
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
Soft-TF (ours)49.28 ± 0.5916.20 ± 1.3361.49 ± 0.9642.06 ± 1.95

We will clarify this in the revision.

Does one strategy work across all datasets?

Continuing from the point above, we show that we can also simply use the same hyperparameters across datasets, and still observe benefits of Soft-TF.

For instance, we show below results with alpha sampled per token for all datasets, and full probability vectors used to construct the soft tokens.

MethodGSM8KMATHMathQAAQuA
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
Soft-TF48.52 ± 1.2415.00 ± 1.2760.55 ± 0.5342.06 ± 1.95

We also show results with alpha sampled per sequence for all datasets, and top-5 probabilities used to construct the soft tokens:

MethodGSM8KMATHMathQAAQuA
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
Soft-TF48.46 ± 1.1016.20 ± 1.3361.23 ± 0.6439.21 ± 2.24

 

"CoCoNut-dagger decoding strategy also vary across datasets or tables"

It is true that the sampling strategies vary per dataset in Table 1. This is intentional, we wanted to show the best performance of CoCoNut-dagger. As shown in Table 10 (Appendix B2), we would have had a weaker Coconut baseline if we used the same sampling method across all datasets. All the other experiments in the paper use standard greedy decoding for sampling.

For ease of reviewing, we also reproduce Table 10 from Appendix B2 below:

MethodGSM8KMATHMathQAAQuA
TF (no CoT)14.00 ± 0.2812.40 ± 0.3249.92 ± 0.4940.71 ± 1.94
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
CoCoNut† (hybrid)20.68 ± 1.710.52 ± 0.440.96 ± 0.5111.89 ± 4.48
CoCoNut† (standard, γ = 0)21.00 ± 1.250.92 ± 0.770.94 ± 0.6911.18 ± 3.81
CoCoNut† (mixed, γ = 0.5)1.00 ± 0.380.04 ± 0.0912.14 ± 2.2812.05 ± 2.58
CoCoNut† (soft, γ = 1)1.00 ± 0.380.04 ± 0.0912.14 ± 2.2812.05 ± 2.58

 


*The held-out validation sets consist of 473/7,473 samples from the GSM8K training set, 500/ 12,000 from MATH, 837/29,837 from MathQA, and 1,467/97,467 from AQuA. We chose these sizes such that we keep round numbers for the training sets.

AuthorsOpenReview ↗

Response to Reviewer UdHv (3/3)

Changing the budget alone moves the Gemma3-1B/GSM8K gain between -0.53 and +3.75 points.

In short, the training budget is a hyperparameter in our setting, and each method is trained for the number of iterations that works best for it on the validation set, and Table 7 shows that teacher forcing doesn’t catch up with Soft-TF when trained for longer.

We chose the number of iterations for each method with a hyperparameter sweep on a held-out validation set. In all but two experiments, the sweep selects the same budget for TF (CoT) and Soft-TF. The two exceptions which are listed in Appendix A, are Gemma3-1B on GSM8K (the case raised by the reviewer) and Gemma2-2B on MATH with top-5 tokens. In these two cases, the sweep selects more iterations for Soft-TF (10k) than for TF (5k). The reviewer is right that the gain then depends on the budget: it ranges from -0.53 (when Soft-TF is trained for TF’s budget) to +3.75 (at Soft-TF’s own budget).

This raises the question whether Soft-TF is better simply because it trains longer, and the answer is no, with evidence from Table 7 that shows that no budget lets TF match Soft_TF.

 

α\alpha granularity and Bernoulli

In short, α\alpha granularity is a hyperparameter that we select using the validation sets, and matching the granularity for Bernoulli rows of Table 4b shows that Soft-TF with Beta is still better.

For all the experiments in the paper (including Tables 2-6 and Fig.2), Soft-TF samples alpha at each iteration per token for GSM8k/AQuA and per sequence for MATH/AQuA. This is an experimental choice that is validated using held-out validation sets. However, in Table 4b, we adopted by default Bernoulli replacement for GSM8K and MATH. To match Beta and isolate the possibility of the improvement coming from different sampling granularity, we rerun Bernoulli by using one draw per sequence for MATH. Using the full probability vectors gives 13.76±1.11 and using the top-5 values gives 14.12±1.36. The difference holds, so the effect is not only explained by the granularity of alpha sampling. We will add these results. For clarity, we reproduce our tables with these results below:

Soft tokensWeight distributionGSM8KMATH
–TF (CoT)44.70 ± 0.8914.44 ± 0.91
FullBeta48.52 ± 1.2415.48 ± 1.50
FullUniform47.61 ± 1.0415.12 ± 0.94
FullBernoulli47.92 ± 1.0113.76 ± 1.11
Top-5Beta49.28 ± 0.5916.20 ± 1.33
Top-5Uniform47.52 ± 0.6613.60 ± 2.02
Top-5Bernoulli48.63 ± 0.6014.12 ± 1.36

 

Answer-format failures rates and confirmation that gamma is applied.

Thank you. As shown below, the answer-format failure rates are indeed high on MathQA and AQUA. We confirm that γ\gamma is applied for both mixed (γ=0.5\gamma=0.5) and soft (γ=1\gamma=1) decoding. The identical results hint that using hidden states for reasoning determine the reasoning path which collapses.

Answer-format failure rates:

DecodingGSM8KMATHMathQAAQuA
Hybrid35.03 ± 2.4647.44 ± 1.9297.45 ± 1.4169.06 ± 7.45
Standard34.97 ± 2.2546.04 ± 2.5697.34 ± 1.6769.45 ± 6.19
Mixed31.14 ± 9.6391.60 ± 1.8254.47 ± 8.1553.07 ± 10.31
Soft31.14 ± 9.6391.60 ± 1.8254.47 ± 8.1553.07 ± 10.31

 

Four-dataset comparison with Llama3.2-3B and full fine-tuning runs.

We run Llama3.2-3b on the 4 datasets. To show the transferability of hyperparameters, we don’t perform sweeps but adopt the same hyperparameters previously used for Gemma2-2B. The results below show that Soft-TF consistently improves upon standard SFT across each dataset. For full fine-tuning, we did 3 runs. The performance is generally lower than LoRA for GSM8K, MATH and MathQA, and we believe this is due to overfitting since the datasets are relatively small. This is supported by our results: for AQuA, which has the largest training set, the results are better than LoRA. For MathQA with full fine-tuning, Soft-TF doesn’t improve SFT, though we believe this can be remedied by addressing the overfitting, and with hyperparameter tuning.

Fine-tuningMethodGSM8KMATHMathQAAQuA
LoRASFT48.58 ± 1.2011.76 ± 0.8365.14 ± 0.9638.19 ± 2.35
LoRASoft-TF52.66 ± 1.1212.64 ± 0.6266.04 ± 0.2640.03 ± 0.91
FullSFT33.05 ± 0.427.93 ± 0.8160.98 ± 0.6041.86 ± 2.17
FullSoft-TF34.90 ± 0.658.27 ± 0.9960.60 ± 0.3543.31 ± 3.36
Reviewer_bAWKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Soft-TF, using soft tokens to improve SFT instead of replacing discrete chain-of-thought. A non-gradient teacher-forced pass produces the model’s predicted distribution at each position, which is projected through the embedding matrix into soft tokens. Training inputs are random convex mixtures of ground-truth embeddings and these soft tokens (α\alpha ∼ Beta(0.1, 0.1)), while the loss and inference stay unchanged. Across 7 LoRA-finetuned models on GSM8K, MATH, MathQA and AQuA, Soft-TF improves over standard SFT by about 1.6–4.6 points.

Critical strengths
  1. Simple and reasonable method: Soft-TF adds no other parameters and leaves inference untouched. It improves over standard TF (CoT) consistently across models and datasets.
  2. Extensive experiments and ablations: The paper covers 7 models and 4 math datasets with 5 seeds. Ablations on soft-token type, number of passes, α\alpha distribution and inference strategy, together with comparisons with label smoothing and noisy embeddings, support the design choices.
Critical weaknesses
  1. Limited methodological novelty. Soft-TF builds closely on the two-pass, probability-weighted embeddings of Mihaylova & Martins (2019), with the main change being random interpolation instead of scheduled replacement. The comparisons support this change empirically, but provide limited analysis into when and why it helps.
  2. Unclear model-selection protocol. Appendix A selects each method’s training budget by the highest mean accuracy over five seeds, but does not identify a separate validation split. If the reported test sets were used for selection, the results would be subject to selection bias.
  3. Unexplained sampling granularity. α\alpha is sampled per token on GSM8K and AQuA but per sequence on MATH and MathQA (Appendix A). The paper does not explain the reason for this choice or show whether one strategy works across all datasets.
Questions for authors
  1. What is the average response length in each training dataset? Do you think Soft-TF helps reduce exposure bias, and might this benefit be greater for longer outputs?
  2. Why is α\alpha sampled per token on GSM8K and AQuA but per sequence on MATH and MathQA (Appendix A)?
  3. Could you give more analysis of why interpolation works better than scheduled replacement?
3 comments
AuthorsOpenReview ↗

Response to reviewer bAWK (1/2)

Methodological novelty

In short, Soft-TF and Mihaylova & Martins (2019) share some methodological aspects but start from opposite motivations, and new experiments show that Soft-TF’s gains come from random interpolation instead of larger soft-input weight.

We agree that Soft-TF is closely related to Mihaylova & Martins (2019), which we discuss in Sections 3.2 and 4 and compare against in Table 1. The key difference is the motivation which leads to the difference in design choice. Mihaylova & Martins’ motivation is to fix the train-test discrepancy: training with teacher forcing causes exposure bias since sampling is autoregressive, so following scheduled sampling, they progressively replace “gold tokens” (ie the ground truth) with model predictions. In their setting, probability-weighted embeddings are only one among several replacement variants they consider. We start from a nearly opposite observation: latent reasoning methods fail because they deviate too far from teacher forcing (Sections 2.2, 3.2, 3.3). Soft-TF uses soft tokens for their expressivity while keeping training close to teacher forcing, which is why we interpolate randomly rather than using a schedule that progressively departs from teacher forcing. Table 1 empirically shows the advantage of Soft-TF.

We agree that more analysis of why and when it helps would strengthen the paper. Following a suggestion from Reviewer 5tLG, we ran a new comparison.

One factor that changes between Soft-TF and scheduled sampling is how much of each method’s input comes from the model’s own predictions rather than ground-truth tokens. In Soft-TF, each input mixes the ground-truth token and the model’s soft prediction whose weight is drawn from a Beta distribution with mean 0.5. In the scheduled sampling baseline, model prediction usage grows over training but averages only 0.25 (See appendix B: the probability of teacher forcing decreases from 1 at the beginning of training to 0.5 at the end). Soft-TF’s advantage could therefore simply come from using more soft tokens. To falsify this hypothesis, we rerun scheduled sampling so that the probability of using ground-truth tokens decays linearly from 1 to 0, which also gives an average of 0.5. We also make the mixing the same as Soft-TF (per token/per sequence). We call this variant SS-matched. The remaining difference from Soft-TF is how soft tokens are used: random interpolation vs replacement based on a schedule.

Results:

MethodGSM8KMATHMathQAAQuA
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
SS46.38 ± 1.0215.04 ± 1.7659.78 ± 1.0139.45 ± 2.32
SS-matched46.51 ± 0.8914.12 ± 1.5159.98 ± 0.5239.84 ± 1.56
Soft-TF49.28 ± 0.5916.20 ± 1.3361.49 ± 0.9642.06 ± 1.95

Matching the amount of soft tokens does not help scheduled sampling, and Soft-TF still outperforms it on all four datasets. Thus, the gain comes from random interpolation not from larger soft-input weight. We will add this new experiment to the revision.

 

Why does interpolation work better than scheduled replacement?

Scheduled sampling replaces the ground-truth tokens with soft tokens, so it is a “hard” replacement. In contrast, interpolation keeps every position at distance α(1−p(y))\alpha(1-p(y)) from the Teacher Forcing (TF input) (in total variation) and preserves the ground truth everywhere. In addition, using a schedule means the TF signal decreases with iterations while soft tokens usage increases. So training ends at a point which is the farthest from TF, and our analysis identifies increased deviation from TF as the failure mode of latent reasoning (Table 1, Figure 2, Table 10).

AuthorsOpenReview ↗

Response to reviewer bAWK (2/2)

Model-selection protocol

In our work, we wanted to have the strongest results for both our method and baselines. We felt that this would be the fairest, since different methods can require different hyperparameters. To this end, we constructed a small held-out validation set* from the training set of each dataset. We then chose hyperparameters by greedily ablating each one whilst keeping others constant, to avoid combinatorial explosion of hyperparameters. We show examples of these sweeps below for GSM8K and MATH with results on the held-out validation sets.

Training budget

IterationsGSM8K SFTGSM8K SSGSM8K Soft-TFMATH SFTMATH SSMATH Soft-TF
1k39.3235.7335.94–––
5k41.2342.4943.7615.4015.2013.60
10k35.3136.7940.3813.6015.8016.60
15k–––11.6012.6011.80

Top-5 vs. full distribution

SettingGSM8K SSGSM8K Soft-TFMATH SSMATH Soft-TF
Top-542.4943.7615.8016.60
Full41.8642.9215.0016.00

Per-token vs. per-sequence

SettingGSM8K SSGSM8K Soft-TFMATH SSMATH Soft-TF
Per-token43.5547.9913.4013.80
Per-sequence42.4943.7615.8016.60

Using the best hyperparameters from the validation set above, we trained on the full training set and evaluated on the test set. This led to the final results, which we include below again for clarity.

MethodGSM8KMATHMathQAAQuA
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
SS (Mihaylova & Martins, 2019)46.38 ± 1.0215.04 ± 1.7659.78 ± 1.0139.45 ± 2.32
Soft-TF (ours)49.28 ± 0.5916.20 ± 1.3361.49 ± 0.9642.06 ± 1.95

We will clarify this in the revision.

 

Does one strategy work across all datasets?

Yes. Continuing from the point above, we show that we can also simply use the same hyperparameters across datasets, and still observe benefits from Soft-TF.

For instance, we show below results with alpha sampled per token for all datasets, and full probability vectors used to construct the soft tokens.

MethodGSM8KMATHMathQAAQuA
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
Soft-TF48.52 ± 1.2415.00 ± 1.2760.55 ± 0.5342.06 ± 1.95

We also show results with alpha sampled per sequence for all datasets, and top-5 probabilities used to construct the soft tokens:

MethodGSM8KMATHMathQAAQuA
TF (CoT)44.70 ± 0.8914.44 ± 0.9159.86 ± 0.8038.19 ± 4.30
Soft-TF48.46 ± 1.1016.20 ± 1.3361.23 ± 0.6439.21 ± 2.24

 

α\alpha sampling granularity.

This is a purely experimental choice which we validated using the held-out validation sets as stated above, and as mentioned above, a single strategy still yields improvements compared to SFT with teacher forcing.

 

Average response length in each training dataset, and does Soft-TF help reduce exposure bias, and might this benefit be greater for longer outputs?

Thank you for this suggestion. There is no apparent correlation between the benefit of Soft-TF and longer outputs.

Average length of generated answers for each dataset:

MethodGSM8KMATHMathQAAQuA
TF (CoT)135.30403.9498.58142.81
Soft-TF134.05356.1399.71119.93

We observe there is no apparent correlation between the benefit of Soft-TF and longer outputs. This supports our perspective about soft tokens adding expressivity, which does not contradict the exposure bias reduction effect. But we believe that if exposure bias was the only problem, we would expect a positive correlation between longer outputs and better improvement, which is not the case.

 


*The held-out validation sets consist of 473/7,473 samples from the GSM8K training set, 500/ 12,000 from MATH, 837/29,837 from MathQA, and 1,467/97,467 from AQuA. We chose these sizes such that we keep round numbers for the training sets.

Thanks for the detailed additional experiments and explanations. These address my major concerns about the model-selection protocol and the choice of α\alpha. However, I still think that the methodological change is largely a modification of the mixing rule within an existing framework, with limited broader insight beyond the performance gains. I therefore maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and paper revision

We would like to thank the reviewers for their feedback. To reflect the discussion in the paper, we have made the following changes in the revised paper, which are highlighted in blue:

  • We clarified the hyperparameter selection strategy and show results on the held-out validation sets (Appendix A, Tables 7, 8 and 9).
  • We added the new experiments showing that a fixed set of hyperparameters across all datasets can still work for our method (Appendix A, Tables 13 and 14).
  • We added to Table 1 a variant of scheduled sampling with matched soft-input weight and describe it in Appendix B.
  • We updated Table4b with Bernoulli results using matched α\alpha sampling granularity for the MATH dataset, and added a clarification about it in Lines 337-338 and Lines 400-401.
  • We show the answer-format failure rates of CoCoNut$^\dagger$ in Appendix B.2, Table 18.
  • We softened claims about latent reasoning requiring compression in the main paper (Lines 144-149) and removed this claim from CoCoNut$^\dagger$ discussion (Lines 302-305).
  • We show additional results with Llama3.2-3B across 4 datasets, using the same hyperparameters as Gemma2-2B showing their transferability across models (Appendix A, Table 16).
  • We show the average response length in Appendix A, Table 15.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.