acceptodds
Under review as a conference paper at ICLR 2027

Towards Understanding the Benefits of Scaffolds: Computational Separations

Abstract

Inference-time harnesses, i.e., external procedures that call a model many times and keep state across calls, can substantially improve model capabilities. In this paper, we investigate the benefits of using such harnesses and the extent to which these benefits can be internalized through training. More precisely, we conceptualize a harness as an arbitrary wrapper around a base model and attempt to distill a student model from the harness's traces. Empirically we find that, even with simple scaffolds and small models, the inference-time harness significantly outperforms a distilled student. We then prove that such a gap is due to a fundamental representational limitation: inference-time harnesses can compute a closure of their per-call primitive that is -complete even when the primitive is a trivial lookup, and that no fixed-depth forward pass can represent under the widely believed conjecture that . Our theory suggests that expanding the representational capacity of the student relative to the task at hand, e.g., by enlarging the model, making the task easier, or lengthening its chain-of-thought, is necessary to close the gap, which we verify empirically. Finally, we scale to an agentic harness on competition math, where the same gap reappears. Overall, our results show that a harness confers on the base model a profound gain in representational power, one that training a same-class student on its traces cannot transfer back into the weights.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Harness, Scaffold, Representational Power, Computational Complexity

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_WhrEOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This paper studies if models can learn inference-time scaffolds from the execution traces of the model using the scaffold. Across tasks like Dyck, Sokoban and competition mathematics, student models that are the same size as teacher models generally underperform the teacher model using the scaffold. The paper argues that larger students, simple scaffolds and longer reasoning chains can help reduce the gap between the student and teacher model. The paper also provides theoretical contributions stating that some scaffold capabilities cannot be transferred into a student with a fixed depth and reasoning length through the graph-search example.

Critical strengths
  1. The paper showcases that a student model, that is equal in size to the teacher, is generally not able to match the performance of a teacher model + scaffold, and this is a quite important finding as this continues to establish why scaffolds are important.
  2. The paper does extensive ablation experiments on different configurations, such as bigger students, simpler scaffolds and longer chain-of-thoughts, that can reduce gap between student and teacher+scaffold.
Critical weaknesses
  1. The paper provides very little information on what specifically the student fails with. It is unclear if there are specific operations the student struggles with or if these errors accumulate over long traces.
  2. The paper provides a decent number of tasks (Dyck, Sokoban, Modular Arithmetic, DAPO-Math-17k, Math Proofs), but its main study with pre-trained model distillation focuses on mathematic tasks, so adding more variety of tasks could provide a more comprehensive picture.
Questions for authors
  1. Could you provide some analysis on specifically what the student fails on for a task? What intermediate steps is the student failing on or do errors accumulate over longer traces?
  2. Do you have distillation results with more pre-trained models and more variety in the tasks besides just math? It would be interesting to see whether a similar student and teacher gap appears in coding benchmarks like HumanEval [1] and more models like Qwen 2.5 or Llama series models.

I lean towards a weak accept because of variety of experiments and theoretical contribution towards understanding scaffolds. Intermediate steps error analysis that clarifies why the students fail would increase my confidence and raise my score.

References

[1] Chen M., Tworek J., (Open AI Team)..., "Evaluating Large Language Models Trained on Code". https://arxiv.org/pdf/2107.03374.

2 comments
AuthorsOpenReview ↗

**Response to Reviewer WhrE**

We thank the reviewer for their time and for the positive comments on our main finding and on the range of our ablations. We address each point below.

W1 & Q1

What intermediate steps is the student failing on or do errors accumulate over longer traces?

Thank you for this suggestion. For the experiments in our paper, we analyzed which intermediate steps the student fails on and whether its errors accumulate.

Sokoban with A*. The student often fails at the step where, at each expansion, it has to work out in place of the base model which two moves are the best legal ones. Errors do accumulate. Before its first error, each expansion fails with a roughly constant probability of about 13%. The teacher needs 28 to 49 expansions, and the chance that the student makes 20 expansions in a row without an error is only about 5%. After the first error, its bookkeeping of the search becomes more and more inconsistent.

Sokoban with VGB. The student often gets the backtracking probability wrong. Errors accumulate here too: once it makes a wrong move, it is much more likely to reach a dead end later.

Modular arithmetic. The student fails on the step that has to evaluate several operators at once. In our setup this is usually the last step, which computes all the remaining operators in one go, so errors do not accumulate.

Math with Branching. Each part the student writes is reasonably good; the problem is that it does not finish. The whole tree is long, and the student fails to write out a tree of that length.

W2 & Q2

Do you have distillation results with more pre-trained models and more variety in the tasks besides just math?

Thank you for the suggestion. To answer your question about other tasks and model families, we ran the following experiment.

We add an experiment showing that the gap also appears on a coding task and with two other model families.

Setup. We use Qwen2.5-1.5B-Instruct and Llama-3.2-1B-Instruct with the same tree as in Section 5 (13 calls). The leaves write code independently, and each node above them keeps a child's program that passes the examples given in the problem statement, asking the model for a new one only if none passes. The hidden tests are never shown to the teacher. The student is the same model with LoRA, trained as in Section 5 on correct teacher traces from MBPP problems 1–500 to write the whole tree in one pass, without an executor. We report the best of four training recipes and evaluate on HumanEval and on the held-out MBPP problems 501–974.

Results (accuracy in %):

ModelBenchmarkTeacherBest student
Qwen2.5-1.5BHumanEval67.443.6
Llama-3.2-1BHumanEval47.422.0
Qwen2.5-1.5BMBPP (held out)66.339.8
Llama-3.2-1BMBPP (held out)54.625.0

The gap between teacher and student is 24 to 30 points for both model families, and HumanEval+ gives the same conclusion. Details are in Appendix F.4 (Table 7) of the revision. These results demonstrate that the gap is not specific to math or to Qwen3.

Thank you to the authors for their detailed responses to my questions. I find the analysis on why the student fails quite interesting and it strengthen's the author's argument that their is a true representational limitation that is preventing the student to match the levels of the teacher. The additional experiments on HumanEval and MBPP generalize these results and reiterate that this gap occurs across various domains and models. Overall, the additions strengthen the author's claims and I will increase my score to a 4.

Reviewer_E3r9OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether inference-time harness gains can be distilled into a student model. The main theoretical result separates an external BFS harness using AC0 queries from a restricted TC0 student, assuming NL is not contained in TC0.

Critical strengths

The formal assumptions are clearly stated.

The empirical comparisons provide plausible and no-surprising observations.

Critical weaknesses
  1. A largely constructed computational separation with insufficient novelty and relevance to ICLR.

Theorem 1 essentially places a polynomial-time BFS algorithm outside a deliberately restricted student complexity class. The base model merely performs deterministic adjacency lookups; the entire computational advantage comes from external BFS. The separation follows directly from classical NL-completeness and existing short-CoT expressivity bounds (Merrill & Sabharwal, 2025).

Replacing the language model with a trivial lookup routine would leave the theorem unchanged. This raises a fundamental question: what has actually been learned about language models or harnesses? The result neither characterizes when a harness improves a particular model nor provides actionable principles for designing effective search, decomposition, or feedback. I find this primarily an artificial worst-case complexity construction, with insufficient new explanatory value for the ICLR audience.

  1. Missing probabilistic learning mechanisms and an unsubstantiated connection between theory and experiments.

Real harnesses depend critically on the model's conditional distribution pθp_\theta, on-policy generation, model-specific error patterns, and feedback-driven adaptation. These mechanisms are central to probabilistic backtracking (Rohatgi et al., 2026) and evolving heuristic systems (Weng, 2026). Although the paper's VGB experiments involve stochastic proposals, its central theorem completely removes these mechanisms. It therefore cannot explain why certain harnesses work for some models but fail for others, or how useful rewards and adaptation strategies emerge.

Moreover, the empirical distillation gap is not evidence of the proposed representational impossibility. The teacher benefits from external search and verification, while the student does not. Larger students performing better does not distinguish expressivity from optimization, distribution mismatch, or inference-compute differences. The theorem's O(log⁡n)O(\log n)-CoT restriction also does not correspond to the much longer autoregressive traces in the experiments.

Overall, I do not find the proposed theoretical explanation convincing or sufficiently connected to the actual learning problem.

Questions for authors

What genuinely new, model-specific insight does Theorem 1 establish beyond classical circuit-complexity separation? Can it predict any observed harness behavior?

What evidence rules out optimization, distributional mismatch, and missing online feedback as explanations for the teacher–student gap? How does the theory account for model-dependent harness effectiveness?

References

Merrill & Sabharwal — A Little Depth Goes a Long Way: The Expressive Power of Log-Depth Transformers.

Rohatgi et al. — Taming Imperfect Process Verifiers: A Sampling Perspective on Backtracking.

Weng — Learning Beyond Gradients.

4 comments
AuthorsOpenReview ↗

**Response to Reviewer E3r9 (1/3)**

We thank the reviewer for their time and effort in reviewing our paper. We address each concern below.

W1

A largely constructed computational separation with insufficient novelty and relevance to ICLR.

The purpose of the theory is explanatory rather than to develop new technical machinery. We agree that much of the theory is technically elementary and combines known results. As we say in the introduction, our broader goal is to provide different lenses for understanding the possible benefits of harnesses—and for this purpose, established results may already do the job when viewed in the right light. We view our paper as providing such a lens for computational benefits.

The construction is worst-case. Restricting the expressive power of the student does not, by itself, make the construction artificially worst-case. In some sense, any computational separation between classes of models has to specify their computational resources: if both classes are allowed unrestricted computation, there would be nothing to establish.

Hence, the relevant question is whether these restrictions are natural and meaningful: by our lights, we use standard restrictions motivated by the expressive power of fixed-depth Transformers. Indeed, TC0\mathsf{TC}^0 is the standard computational abstraction for fixed-depth Transformers, as log-precision transformers can be simulated by TC0\mathsf{TC}^0 circuits (Merrill et al., 2022; Merrill & Sabharwal, 2023), and many works on the expressive power of transformers and chain of thought use the same model (Li et al., 2024; Merrill & Sabharwal, 2024; Strobl et al., 2024).

Principles for designing harnesses. We agree that the theorem doesn't characterize what are better search, decomposition, or feedback strategies. The goal is to isolate a more basic computational benefit: even a simple harness can expand what a model can compute by organizing repeated calls in a harness, and this benefit may not be reproducible by changing the weights of a student with fixed computational resources (size + chain of thought length). This has practical implications for distillation: one must also expand the student's computational resources. Our experiments then provide case studies of this phenomenon in different settings.

References

  • Strobl, L., Merrill, W., Weiss, G., Chiang, D., and Angluin, D. What formal languages can transformers express? A survey. TACL 2024.
AuthorsOpenReview ↗

**Response to Reviewer E3r9 (2/3)**

W2

Missing probabilistic learning mechanisms and an unsubstantiated connection between theory and experiments.

We study a harness without an external verifier or search. The Branching harness in Section 5 has no external verifier and no search, and the gap between the harness and the student is still there.

The teacher's benefit could come from the extra inference-time compute that the harness brings. We agree, this is the point the paper makes! Although the harness itself is simple, it gives the base model effectively more computation, and this raises its expressive power. A student with limited expressive power cannot internalize this benefit.

Extensive ablations suggest optimization is not the main cause. Of course, just empirical studies cannot prove a negative (i.e. “no optimization procedure works”). However, we provide extensive ablations to provide strong evidence that neither optimization nor sample complexity are the main cause. In Section 2.2, Appendix B.3 and Appendix E.1, more training data, more training iterations, a higher LoRA rank (up to full fine-tuning) and other training hyperparameters (the clean-data ratio and the number of epochs) all fail to close the gap. This suggests that the gap is not caused by optimization, training dynamics, or sample complexity. In contrast, a larger model (Section 3.1) or a longer chain of thought (Section 3.3) does close it, which points to expressive power.

Our experiments are in line with the O(log⁡n)O(\log n) CoT restriction. Our theory shows that a student with a short chain of thought cannot internalize the harness; once the chain of thought is long enough, this obstruction no longer applies. The experiment in Section 3.3 (Figure 3) confirms this: with a short chain of thought there is a large gap between the model with and without the harness, and with a long chain of thought the gap closes.

Probabilistic vs. deterministic modeling. We agree that probabilistic mechanisms are important, and we hope subsequent theory will incorporate them. Our starting point is the (deterministic) computational-complexity lens via circuit complexity, which is most commonly used to study Transformer expressivity. Extending this lens to probabilistic harnesses requires further modeling choices—for example, how to capture the model's conditional distributions, adaptation to feedback, etc.—for which there is not yet a clear canonical abstraction. We view our deterministic model as a natural first step, rather than a complete account of how harnesses work.

Q1

What genuinely new, model-specific insight does Theorem 1 establish beyond classical circuit-complexity separation? Can it predict any observed harness behavior?

This is a fair question!

We agree that much of the theory is technically elementary and combines known results. We view this as a feature, rather than a bug! As we say in the introduction, our broader question is to provide different lenses for understanding the possible benefits of harnesses—and for this purpose, established results may already do the job when viewed in the right light. We view our paper as providing such a lens for computational benefits: namely even a simple harness can expand what a model can compute by organizing repeated calls in a harness, and this benefit may not be reproducible by changing the weights of a student with fixed computational resources (size + chain of thought length). The theory therefore provides a lens for understanding when harness benefits may resist distillation, even though it does not characterize how to design search, decomposition, or feedback strategies in the harness for a particular model.

AuthorsOpenReview ↗

**Response to Reviewer E3r9 (3/3)**

Q2

What evidence rules out optimization, distributional mismatch, and missing online feedback as explanations for the teacher–student gap? How does the theory account for model-dependent harness effectiveness?

These are important alternative explanations, and we thank the reviewer for raising them.

Offline distillation is a reasonable choice here (Foster et al., 2024). Foster et al. (2024) show that when the student's model class contains the teacher's policy, behavior cloning with the log loss, which is what next-token training on traces does, does not suffer from errors that compound over the horizon. They also show that, without further assumptions, online feedback cannot improve on it. So, under their assumptions, if the gap came from distribution mismatch or from missing online feedback, the student would have to be unable to represent the teacher, which is a question of expressive power. We explain this choice of offline distillation in our response to Reviewer 1CwZ (W1), and we will add this discussion to the paper.

We added an on-policy distillation experiment, and it does not close the gap. To test whether distribution mismatch or missing online feedback explains the gap, we train the student on its own generations with the teacher's corrections. On Dyck (Section 2), we start from the same-size student of Figure 1a and train it with DAgger: the student writes its own continuations, the teacher (the base model with VGB) gives the correct action at every state the student reaches, including a backtrack where the student has made an error, and we train on these corrections. After three rounds, the student is still far from the teacher out of distribution (accuracy in %):

IDOOD
Teacher (base model + VGB)100100
On-policy distillation (DAgger)99.273.2

Details are in Appendix F.1 (Table 4) of the revision. These results demonstrate that on-policy feedback does not close the gap.

Our ablations also make optimization an unlikely cause. More training data, more training iterations, a higher LoRA rank (up to full fine-tuning) and other training hyperparameters do not close the gap, while a larger model or a longer chain of thought does (see our response to W2).

We are not trying to explain all benefits of harnesses, only to isolate one computation-centric benefit. We agree that our theory does not explain why a harness helps some models more than others. This is beyond the scope of the theory, whose goal is to explain why a harness increases expressive power: the reason is not that the harness itself is very expressive, but that the extra computation from many calls raises what the model can compute.

References

  • Foster, D. J., Block, A., and Misra, D. Is behavior cloning all you need? Understanding horizon in imitation learning. NeurIPS 2024.

Thank you again for your time and valuable feedback. As the discussion period is coming to a close, we would like to kindly check whether our rebuttal has addressed your concerns. We would be happy to clarify any remaining questions.

Reviewer_1CwZOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the role of harnesses in modern agentic settings and asks how much harnesses improve the capability of their base models. It also asks if these abilities (e.g., searching) can be internalized by the models. To see this they try to see if the trajectories of a harness + model can be distilled into a similar model. The paper includes experiments on Dyck languages and Sokoban games with search algorithms as well as modular arithmetic, showing that such distillation is often not possible unless a larger student model or a longer chain-of-though is used. Interestingly, we can see that the OOD performance of the student model is particularly worse.

The paper also shows a theoretical result based on the graph reachability task. It shows that there are simple model and harness combinations that cannot even be expressed by the same model based on complexity theory results and assumptions. This result may break however if one uses polynomially large chain-of-thought.

Finally the paper focuses on a more realistic setting trying to solve math problems where it is demonstrated that if harness is complex enough, its performance gain may not be reachable with the student model alone.

Critical strengths
  • The paper studies a timely question, asking about the importance of harnesses and the necessity of using them. The paper starts with puzzle tasks with clear complexity knobs and later provides experiments on more realistic settings as well as theoretical results.
  • The paper is very well-written. For example when using complexity classes it explains them properly.
  • What I personally liked is that the authors pay special attention to OOD generalization and separate it from in-distribution generalization.
Critical weaknesses

There are several points which can make the paper better:

  • The paper currently only considers teacher-forcing (SFT-style) distillation. However, this format can be unnatural for the model. On-policy distillation methods would let the model rewrite the answers in a language that is potentially better for learning or potentially use longer CoT. In particular, the paper doesn't have an experiment showing that distillation works if the agent uses more tokens to explain the harness's behavior. Current token-scaling experiment is not performed in the distillation setting.
  • The paper can be more upfront about the differences between its theoretical result and empirical results. In particular the theory result only holds for logarithmic length CoT.
  • The last part of the paper which is on math competition problems is indeed the most interesting part from a practical perspective. Yet, it's rather short and also uses a very small Qwen3 1.7B base model in non-thinking mode.
  • See below for some other specific points.
Questions for authors
  • Q1. How can the fairness of the proposed harnesses be justified? More specifically, most of them use a verification or scoring function which can be viewed as an external source of information. How can we account for the role of this additional information available only in harness settings? How would harnesses without a verifier or students with a verifier perform?
  • Q2. In the writing of the paper it is sometimes implied that the students cannot internalize due to expressivity problems (e.g., when mentioning more expressive and larger models can learn). Do we have any evidence that let us separate expressivity and learnability?
  • Q3. When increasing the model size, is it the depth that is increased or width? I believe depth targets expressive more directly so having separate experiments for scaling width and depth could be useful.
  • Q4. What is the test math dataset used? Is it the DAPO dataset?
  • Q5. The math experiments are currently conducted using the Qwen3 1.7 base model in non-thinking mode. Why have you chosen such a small model? What would happen with larger, instruct-tuned model in thinking mode. Can be still observe the same phenomenon?
  • Q6. I didn't understand what message the QED-Nano part aims to convey. Can you elaborate please?
  • Q7. Can you provide examples of the traces used in the distillation? Do they include the search keywords?
4 comments
AuthorsOpenReview ↗

**Response to Reviewer 1CwZ (1/3)**

We thank the reviewer for their time and careful reading, and for the kind words about the question we study, the writing and our attention to OOD generalization. We address each point below.

W1

The paper currently only considers teacher-forcing (SFT-style) distillation. [...] the paper doesn't have an experiment showing that distillation works if the agent uses more tokens to explain the harness's behavior.

We focus on SFT distillation on purpose. It is the simplest form of distillation, and when the student can represent the teacher, it is provably optimal (Foster et al., 2024). A gap that remains after SFT therefore points to the student's capacity rather than to the training method, which is what we want to measure. We will make this motivation explicit in the revision. That said, we agree that other forms of distillation are worth checking, so we ran the two experiments below.

We added an on-policy distillation experiment, and it does not close the gap. To answer your question about on-policy distillation, we train the student on its own generations. On Dyck (Section 2), we start from the same-size student of Figure 1a and train it with DAgger: the student writes its own continuations, the teacher (the base model with VGB) gives the correct action at every state the student reaches, including a backtrack where the student has made an error, and we train on these corrections. After three rounds, the student is still far from the teacher out of distribution (accuracy in %):

IDOOD
Teacher (base model + VGB)100100
On-policy distillation (DAgger)99.273.2

Details are in Appendix F.1 (Table 4) of the revision. These results demonstrate that on-policy distillation does not close the gap either.

We add an experiment showing that a longer response in distillation helps. To answer whether distillation works when the student uses more tokens to explain the harness's behavior, we let the student write out what the verifier computes at each step. On the same Dyck task, we train two same-size students on the same VGB traces and change only what the traces contain. In the original format, each step is a score token followed by an action. In the longer format, each action is followed by three extra tokens that write out the state the verifier relies on: the current depth, the bracket on top of the stack, and the number of positions left. The test prefixes come without these annotations, so the student first copies the prefix and writes the state after each bracket, and only then continues. This adds about 200 tokens per sequence. With the same training (20K steps, same learning rate), writing out the state brings the student close to the teacher out of distribution (accuracy in %):

IDOOD
Teacher (base model + VGB)100100
Original format99.874.9
Longer format (state written out)99.494.7

Details are in Appendix F.2 (Table 5) of the revision. These results demonstrate that distillation works once the student can use more tokens to carry out the harness's computation, consistent with Section 3.3.

W2

The paper can be more upfront about the differences between its theoretical result and empirical results. In particular the theory result only holds for logarithmic length CoT.

Thank you for pointing this out. We agree that the paper does not say clearly enough that Theorem 1 only covers chains of thought of logarithmic length. We will make this explicit in the next version.

Our experiments also vary the length of the chain of thought (Section 3.3, Figure 3) to confirm the theory. On modular arithmetic, a model trained with a short chain of thought stays well below the harness, while a model trained with a long enough chain of thought matches it. This is in line with the theory.

Q1

How can the fairness of the proposed harnesses be justified? [...] How would harnesses without a verifier or students with a verifier perform?

This is a good point, and it is certainly important to consider what information a harness brings in. We agree that any harness adds something from outside the model. In fact, this is exactly our point: a harness increases the expressive power of the model because it brings in this external information, and a student has to reproduce it inside a single model.

Not all of our harnesses use a verifier. In the math experiments (Section 5), the Branching harness only calls the model along a tree: the leaves solve the problem independently, and each internal node writes a new solution from the solutions of its children. There is no verifier, and the gap between the harness and the student is still there.

References

  • Foster, D. J., Block, A., and Misra, D. Is behavior cloning all you need? Understanding horizon in imitation learning. NeurIPS 2024.
AuthorsOpenReview ↗

**Response to Reviewer 1CwZ (2/3)**

Q2

Do we have any evidence that let us separate expressivity and learnability?

This is an important question, and we answer it in three parts.

We trained for long enough and swept the main hyperparameters. Our training budgets are long enough for accuracy to level off. For the same-size student fine-tuned from the base model, going from 3K to 8K iterations changes its accuracy by only a few points, and twenty times more traces by about half a point (Appendix B.3); on Qwen3-1.7B, training stops improving after two epochs (Appendix E.1). We also swept the clean-data ratio, the distribution of the training problems and the random seed (Section 2.2, Appendix B.3). These extensive ablations suggest that the gap isn't due to training dynamics or optimization.

Expressivity and learnability are hard to separate. It's of course impossible to separate with certainty empirically expressivity and learnability—it's always possible there's a better, non-standard optimizer that can reach a better solution at a particular capacity. That being said, the thorough sweeps over the ingredients of standard optimizers mentioned above provide quite strong evidence that the core issue is expressivity.

Increasing depth helps, while increasing the LoRA rank does not. If the intended comparison is between increasing the depth and increasing the LoRA rank, which is closer to our theory, our results show a clear difference. On Qwen3-1.7B, a higher LoRA rank, and even full fine-tuning, leaves the student below the teacher, while the larger Qwen3-4B, which is also deeper (36 layers instead of 28), passes the teacher (Appendix E.1, Figure 15). In the controlled tasks, adding depth helps on Dyck, where adding width alone stops helping; see our answer to Q3.

Q3

When increasing the model size, is it the depth that is increased or width?

Thank you for this suggestion.

We have separate results for width and depth. Increasing only the width already raises the student's expressive power enough to pass the teacher, and increasing the depth helps further.

In our size sweep (Table 1), the first six sizes keep two layers and only increase the width, and the larger sizes also add layers. On Sokoban with VGB, width alone is enough: a two-layer student passes the teacher once it is wide enough, and adding layers changes little after that. On Dyck with VGB, increasing the width stops helping after a point, while deeper students keep improving out of distribution. This fits the intuition that tracking the bracket stack in Dyck needs depth. Details are in Appendix F.3 (Table 6) of the revision.

Q4

What is the test math dataset used? Is it the DAPO dataset?

Yes. The test set is 500 problems from DAPO-Math-17k, disjoint from the 1,500 problems used to generate the training traces. This is stated in the Setup paragraph of Section 5 (and in the introduction); in the revision, we have made it explicit that the 500 test problems also come from DAPO-Math-17k.

Q5 & W3

Why have you chosen such a small model? What would happen with larger, instruct-tuned model in thinking mode.

We use a small model in non-thinking mode so that the teacher's traces are short enough to train on. With a thinking model or a larger model, the teacher's traces are usually much longer, so training would need far more compute and time and would also be less stable. This is why we use a small model in the paper.

We add an experiment with a larger model. We repeat the experiment of Section 5 with Qwen3-8B instead of Qwen3-1.7B: the same DAPO math problems, the same Branching tree (nine independent solutions, three intermediate aggregations and one final aggregation, 13 calls), temperature 1.01.0, and at most 40964096 tokens per call. The student is the same Qwen3-8B with LoRA, trained with the recipe of Section 5, and may generate up to 32K tokens. Results (accuracy in %):

ModelBaseTeacherStudentTeacher − Student
Qwen3-1.7B (Section 5)29.340.435.25.2
Qwen3-8B51.462.860.02.8

With the 8B model, the gap between the teacher and the student is still there, so our results also hold for a larger model.

AuthorsOpenReview ↗

**Response to Reviewer 1CwZ (3/3)**

Q6

I didn't understand what message the QED-Nano part aims to convey. Can you elaborate please?

We are sorry that this part was unclear. For a model that is already strong, such as QED-Nano, a harness only helps when it is large. But once the harness is large, its trace is too long to fit in one context, so it cannot be distilled into a single rollout. In practice, then, the harness gains that matter for a strong model cannot be moved into its weights by distilling the trace. This further shows that, for practical reasons, we often cannot internalize a harness into the weights. To make this clearer, we will rewrite the QED-Nano paragraph in Section 5 so that it starts with this message, explains why we do not distill this harness, and moves the details of how QED-Nano was trained to Appendix E.2.

Q7

Can you provide examples of the traces used in the distillation? Do they include the search keywords?

Yes. The traces contain explicit search tokens. In VGB, each step is the verifier's score followed by the action or a backtrack token <BT>. In A*, the trace lists every <CREATE> and <CLOSE> event with the node's coordinates and its gg, hh and ff values, and ends with <PLAN> followed by the moves. In Branching, the trace contains each attempt under an ## Attempt header, followed by the final solution under ## Final Solution. The formats are given in Appendix B.1 and Section 5, and we will add a full example of each trace to the appendix.

Thank you again for your time and valuable feedback. As the discussion period is coming to a close, we would like to kindly check whether our rebuttal has addressed your concerns. We would be happy to clarify any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.