- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Best-of-Depth (BoD), a training objective for looped Transformers that treats successive recurrent readouts along a single latent trajectory as a candidate set. Rather than supervising only the final depth (Last-Depth) or averaging losses across depths (Mean-Depth), BoD minimizes the minimum loss across all supervised depths. Without architectural changes, this encourages the model to visit distinct valid solutions across recurrent steps rather than collapsing to a single fixed point. Experiments on synthetic multi-solution tasks (N-Queens completion, graph coloring) and LLM code generation fine-tuning (Huginn-3.5B on HumanEval and MBPP) show consistent improvements in Success@Depth and Coverage@Depth.
Critical strengths
- Simple, Principled, and Well-Executed Idea: Bringing Multiple Choice Learning (MCL) to the recurrent depth axis of looped Transformers is natural and requires zero architectural modifications. The symmetry-breaking analysis (Proposition 3) clearly illustrates why
Last-Depth,Mean-Depth, andSoft-Depthaverage out multi-modal targets whileBoDpreserves distinct solutions along a recurrent orbit. - Insightful Mechanistic & Ablation Studies: The withheld-solution experiment (Sec. 4.5.3, Table 2) and trajectory visualizations (Sec. 5, Fig. 4) convincingly demonstrate that BoD learns exploratory latent dynamics that continue moving across valid solution basins rather than converging to a static fixed point.
- High Reproducibility and Rigor: The experimental controls (isomorphism/symmetry split separation, canonical coloring equivalence classes, temperature sweeps, and complete appendices) are meticulous and transparently documented.
Critical weaknesses
- The mechanism may not transfer directly from multi-target tasks to code fine-tuning. The combinatorial tasks repeatedly pair an input with different valid targets, whereas each CodeFeedback example has one teacher response and is used for one fine-tuning epoch. In this single-target setting, why should BoD induce multiple valid solutions for the same input rather than learn a preferred depth? The one-teacher graph-coloring ablation is suggestive, but does not establish the mechanism for code generation.
- Code generation lacks a compute-matched sampling baseline. The six evaluated depths require separate complete autoregressive generations. Success@32 establishes that at least one candidate passes the tests, but does not show whether depth-wise candidates outperform six stochastic samples at a fixed depth—or how a correct candidate would be selected in practice. Variation across independent fine-tuning runs is also unreported, and MBPP uses prompt-visible public tests.
- LLM training overhead and gradient truncation are insufficiently characterized. BoD scores all 32 depths, recomputes the winning depth, and backpropagates through at most its last eight recurrent updates. The relative wall-clock time and peak memory versus Last-Depth, and whether the two methods use the same truncation policy, should be clarified.
Questions for authors
- At comparable inference cost, how does the six-depth BoD candidate set compare with six stochastic samples from Last-Depth at a fixed depth?
- What evidence distinguishes within-trajectory solution diversity from per-input depth selection under single-target training?
- What are the respective fine-tuning time, peak memory, and backpropagation ranges?
3 comments
Response
Thank you for your explanation. I will keep my rating.
Thank you for reviewing our responses and for your constructive feedback. We appreciate your time and consideration.
Please let us know if any further clarification would be helpful.
We thank the reviewer for the positive assessment and precise questions. We answer each point below.
Code problems do admit multiple valid solutions; what is single in CodeFeedback is the supervision, one teacher response per problem. With one target per example, each training example updates only the depth that fits it best, so different depths are trained on different examples and come to produce different programs, giving complementary candidates for the same problem. The evidence that this is within-trajectory diversity, not only per-input depth selection, is that one trajectory often contains several distinct correct programs for the same problem. After removing docstrings and renaming identifiers, BoD's six depth-wise candidates contain two or more distinct correct programs for 32 HumanEval and 127 MBPP problems, versus 21 and 83 for Last-Depth. If BoD only selected one suitable depth per input, the other depths would not add distinct correct programs. The average number of distinct programs among the six candidates is also highest for BoD (4.54 on HumanEval and 4.45 on MBPP, versus 4.43 and 4.28 for the base model and 4.10 and 3.96 for Last-Depth). BoD also does not settle on one preferred depth: during fine-tuning, the selected depths of the middle 80% of examples span 13–31, and this spread is the same at the start and end of training.
The recurrent block, which runs d times per token at depth d, dominates inference cost. BoD's six candidates (depths 4, 8, 12, 16, 24, 32) therefore cost 4+8+12+16+24+32=96 recurrent steps per token, the same as three samples at depth 32 (3% more counting all layers); six samples would cost twice as much. We compare with three samples from the paper's Last-Depth model at depth 32 (temperature 0.8, top-p 0.95).
Problems solved by at least one candidate (%), equal compute, the paper's models:
At matched recurrent compute, BoD's six depth-wise candidates solve more problems than three samples from Last-Depth on both benchmarks. A correct candidate is selected by running tests, as in pass@k. Prompt-visible tests are the standard MBPP protocol (Austin et al., 2021); HumanEval uses hidden tests and shows the same trend.
Measured back to back on the same 8×H200 node (median over 20 updates):
Both use the same truncation: gradients flow through at most the last eight recurrent updates of the supervised depth (Last-Depth: updates 25–32 of depth 32; BoD: the last min(d, 8) updates of the selected depth d). BoD scores all 32 depths along one trajectory without gradients and keeps the recurrent states. Its only extra work over Last-Depth is therefore the readout at each depth.
We would be glad to clarify anything further.