acceptodds
Under review as a conference paper at ICLR 2027

Allspark: Weak to Strong Transfer via Alternating Chains of Thought

Abstract

Recent progress in frontier models has renewed interest in large-scale reinforcement learning (RL), but the cost of generating large-model rollouts makes even testing RL recipes expensive. We ask whether reasoning improvements learned by a small, weak model can benefit a larger, stronger model without using the strong model's rollouts during training. We introduce Allspark, a training and inference framework for weak-to-strong transfer through alternating chains of thought. A weak teacher is trained alongside a frozen copy of the same model; the two alternate reasoning segments, and the frozen model produces the final answer. At inference time, a stronger student replaces the frozen training partner, while both models remain fixed. Because they communicate through text, the teacher can steer students from different model families and with different tokenizers. We study Allspark at two scales: controlled Qwen experiments across math and reasoning, and larger-scale Inkling experiments on ARC-AGI-2. The Inkling experiments show accuracy gains in within-family and cross-family settings, including transfer to Kimi and Nemotron, with benefits that vary across inference settings. These findings motivate reusing a trained weak teacher across strong students and examining the resulting accuracy–token tradeoff.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Weak to Strong Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2mmYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Allspark, a framework for weak-to-strong transfer through alternating chains of thought. During training, a weak teacher is trained with reinforcement learning alongside a frozen copy of the same model. During inference, the frozen copy is replaced by a stronger student, while both models remain fixed. Experiments on Qwen and ARC-AGI-2 show improvements in several settings, including cross-family transfer.

Critical strengths

The appendix provides lots of case studies across math, reasoning, and coding, illustrating how teacher-generated reasoning affects student outputs.

Critical weaknesses
  1. The paper seems to have been completed in a rush. The main text is seriously insufficient and falls far short of the 9-page limit. Pages 6 and 7 are occupied by three figures, leaving a large amount of blank space.

  2. The authors seem to deliberately blur the distinction between OPD and on-policy self-distillation (OPSD). In this paper, both the student and the teacher are set to the model itself, yet the authors still emphasize “weak teacher” and “strong student.” There is a major logical problem here.

  3. Based on Point 2, the experimental baselines do not include OPSD methods. The number of experiments, the choice of benchmarks, and other aspects are seriously insufficient and unconvincing.

Questions for authors

See Weaknesses.

Reviewer_duqUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Allspark trains a weak model to collaborate with a frozen partner by alternating reasoning segments. During training, the partner is a frozen copy of the weak model's initial checkpoint and produces the final answer; task rewards update only the teacher's reasoning tokens. At transfer time, the trained teacher is frozen and the partner is replaced with a stronger student. The models exchange reasoning through text, and the student produces the final answer. Transfer therefore denotes a benefit from inference-time collaboration, rather than an update to the strong student's parameters or a persistent improvement in its standalone ability.

The paper evaluates Qwen3-1.7B teachers with Qwen3-4B students on mathematics and logical reasoning, and an Inkling-Small teacher with Inkling, Nemotron, and Kimi students on ARC-AGI-2. The ARC evaluation uses 96 development tasks drawn from the public training split, with three attempts per task, reporting mean single-attempt success rather than best-of-three accuracy. The authors report accuracy improvements across model families and fewer retained tokens in some settings. The appendix also estimates inference costs and explicitly distinguishes token reductions from monetary savings.

Critical strengths
  1. The separation between teacher training and transfer is practically meaningful. Teacher training uses only a weak-model partner, while deployment can pair the trained teacher with different frozen strong students through text. Shared tokenizers and strong-student parameter updates are unnecessary. This removes the strong-student training rollouts required by weak-to-strong OPD. The core approach is simple and is instantiated both with Qwen models and with larger served models.

  2. The cross-family results provide a promising signal, with useful disclosure of failures and costs. Figure 5 reports gains of 17.7, 5.2, and 11.5 percentage points for Nemotron medium, Nemotron full, and Kimi, respectively. Task-level paired bootstrap intervals appropriately account for the task structure rather than treating all 288 attempts as independent tasks. Appendix B.3 includes an untrained-teacher control at an earlier checkpoint, and Appendix C includes a correct-to-incorrect outcome. Appendix B.4 reports that, at equal Inkling effort, modeled cost is 1.42–1.82× the student-alone cost; for Kimi, 7.7% fewer retained tokens accompany approximately 21.1% higher modeled cost. These disclosures make the tradeoffs assessable.

Critical weaknesses
  1. The strongest cross-family gains conflate teacher contributions with changes to the inference procedure. Section 4.3 and Appendix B.2 state that Allspark explicitly closes reasoning and requests a separate answer after stopping, whereas the student-alone baseline must transition to a visible answer within one completion. Thus, even with the same nominal reasoning-plus-answer allowance, the baseline may fail because it never produces the final JSON. This is a valid comparison of complete inference systems, but does not isolate transfer from the trained weak teacher. Controls with the same forced answer stage, student self-continuation, and untrained/ordinary-RL teachers are missing for the final cross-family configurations. The checkpoint-35, effort-0.7, single-attempt control in Appendix B.3 is useful but does not replace those controls at checkpoint 63 and the reported cross-family settings. Retained-token accounting also excludes discarded generated suffixes and therefore does not measure total generation cost, as the authors acknowledge.

  2. Limited independent validation and specific record inconsistencies weaken the small-scale evidence. Qwen Table 1 explicitly states that the same development panel is used for checkpoint selection and subsequent reporting. Only one training seed and one sampled answer per question are used. For the 4B student, mathematics changes from 96.9% to 96.1% and reasoning from 81.2% to 82.8%, approximately one fewer and two additional correct questions, respectively. These changes provide weak evidence of robust positive transfer. More concretely, Table 2 reports 89.8% mathematics accuracy for checkpoint 94 with the 1.7B student, while Appendix Table 4 reports 119/128, approximately 93.0%, for that checkpoint without explaining a protocol difference. The main text and Table 1 describe two domains and 304 prompts, whereas Appendix A.3 specifies six groups with two per domain, Figure 7 includes Code, and Appendix A.6 includes TACO in a three-domain development panel. A consistent training, selection, and evaluation record is needed. The ARC evaluation also uses public training tasks; choices of effort, checkpoints, and pilot configurations should be separated from final generalization evaluation, and possible pretraining exposure should be acknowledged.

  3. This paper seems to be an autoresearch paper.

Questions for authors
  1. [Primary attribution control] How much of the Figure 5 gain remains if the student-alone baseline receives the same forced separate-answer stage after a fixed reasoning budget, identical JSON instructions, and reserved answer space? Please break failures down into reasoning errors, missing visible answers, formatting failures, and truncation. This evidence would most directly affect the assessment.

  2. [Value of teacher training] For at least one cross-family student, could the final checkpoint, effort, and termination settings be evaluated with an untrained teacher, an ordinary-RL teacher, and student self-continuation? The earlier Inkling control improves from 75.0% to 76.0%, with 66.7% for the untrained teacher. Do the larger final gains persist under these matched comparisons?

  3. [Results and data configuration] Do the 89.8% in Table 2 and 119/128 in Table 4 correspond to different samples, seeds, protocols, or logs? Does Qwen training include TACO/Code? Please specify the actual pool size, domains, and groups per update, and reconcile Appendix A.3, Appendix A.6, Table 1, and Figure 7. The current description does not uniquely determine the configuration.

  4. [Independent evaluation] What checkpoint candidate set and selection criteria were used, and why is ordinary-RL step 92 selected differently from Allspark? Do the ARC checkpoint/effort choices or the 16-task target-model pilot overlap with or depend on the final 96 tasks? Could the authors add evaluation on tasks not used for selection or independent repetitions under a fixed rule, retaining the task-level paired intervals? Separation from this RL training pool does not establish separation from model pretraining.

  5. This paper seems to be an autoresearch paper.

Reviewer_XYryOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Allspark trains a weak teacher by alternating its reasoning with a frozen partner, then pairs the teacher with stronger models at inference. The experiments cover several model families and show promising transfer, with stronger gains on ARC than on the Qwen tests.

Critical strengths
  1. Allspark trains a teacher without requiring training responses from the target strong model.
  2. The evaluation tests transfer across several model families and reports limitations openly.
Critical weaknesses

W1: Differences in answer handling may contribute to the reported transfer gains. Allspark gets a separate answer-generation step, while the student baseline must reason and answer in one completion. W2: Reusing development tasks weakens the evidence for generalization. Some evaluation tasks are also used to choose checkpoints.

Questions for authors

Do the cross-family gains persist when the student-alone baseline uses the same forced closing and separate-answer procedure?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.