acceptodds
Under review as a conference paper at ICLR 2027

Imagine: Selective Width Expansion for Hard-Token Reasoning

Abstract

Small language models can fail at a few difficult predictions and spoil an otherwise correct answer. Extra reasoning steps can repair these errors, but serial computation makes users wait. We introduce Imagine, a post-training method for selective width expansion within a single Transformer forward. Parallel Slot Expansion gives difficult positions persistent expert states. These states interact across layers and merge only at prediction time. A Lookahead Gain Controller allocates width before the current input enters the model. Dense-to-Hard Training teaches the model to use this capacity, then concentrates it on difficult predictions. Our analysis connects the resulting structured wide network to conditions for lower prediction loss. Across three reasoning benchmarks, Imagine+ 0.6B achieves 5.8% higher average accuracy than plain supervised fine-tuning. Parameter-matched variants also improve average accuracy. Moreover, the single-slot model preserves SFT-level single-user throughput and reduces the mean latency of the slowest 1% of tokens by 34.9% relative to parameter-matched recurrent transformer.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Latent thinking, MoE, Language Model, LLM, Pause token

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pkrDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a method to adapt autoregressive transformers to be able to adaptively leverage parallel computation in a manner similar to pause tokens.

  • Leveraging parallel computation: the model forwards the same input embeddings through the transformer in parallel "lanes" where each lane uses a unique LoRA adaptation of the backbone. The primary lane is called the real lane and the optional additional lanes are called slot lanes. A lightweight module is used to combine the outputs from multiple lanes into a single input for the shared LM head. Communication between lanes is done through causal attention between lanes. I.e., slot lanes attend to the KV cache of previous real and slot lanes within the same token.
    • Hidden cost: while real lanes only attend to the KV cache of real lanes in previous tokens, slot lanes attend to not only the real lanes but also the slot lanes (with <= slot index) of previous tokens. Therefore, all slots add KV cache overhead, in addition to linear increase in FFN FLOPs and slightly super-linear cost in attention FLOPs and KV cache reads. These cost factors could be better explained in the paper -- they can be deduced but not pointed out.
  • Adaptive parallelism: the method involves training a decision head (the Lookahead Gain Controller, LGC) based on the output hidden state of the previous step and the current input embedding. The decision head is supervised based on the teacher-forced cross-entropy loss at various slot budgets k.
  • Curriculum: the model is adapted via SFT through two phases: (1) dense-to-hard (DHT) curriculum where the model learns to utilize slot lanes through end-to-end training with dropout-style application of slot lanes and (2) training the LGC for adaptive use of slots.
Critical strengths
  • The motivation for adaptive parallel compute is clear and the methodology nails the execution, i.e., simple, comprehensive, and aligns well with the motivation, down to the details of e.g. handling inter-lane communication/attention.
Critical weaknesses

Overall, the evaluation and analysis is limited in many aspects and does not substantially increase confidence in the viability of pause-token style (adaptive) parallel computation, considering generality, scalability and train/inference cost tradeoffs.

  • It is unclear if the method achieve robust performance gains due to:
    • Limited evaluation on three math reasoning datasets on a single backbone model
    • Sensitivity to threshold hyperparameter: while the accuracy gains seem somewhat significant, it is highly sensitive to the threshold hyper parameter as shown in Figure 4 and 7. At many points, accuracy is lower than the SFT baseline.
  • It is unclear if and how this method scales across model sizes and LoRA ranks. Only a single size 0.6B and LoRA rank (16 per slot) is studied. For context to reader, hidden dim is 1024. Additional scaling studies would be reasonable, and feasible even on a relatively low compute budget. LoRA rank analysis is not critical but would be interesting.
  • Cost tradeoffs are not comprehensively studied: the main cost of this method is additional FLOPs due to parallel slot lanes. As explained in the Paper Summary, each slot incurs additional FLOPs for both attention and FFN operations as well as KV cache overhead. This has implications that have not been addressed in training and inference.
    • Training cost: the paper indicates 1.7x training time compared to regular SFT for the method. Additional experiments accounting for this cost, such as training-cost controlled experiments may be helpful. Note that, I believe a more principled large-scale study would be needed to obtain meaningful signal on whether the additional training cost can be justified in practice, in a similar spirit as pretraining scaling laws. I don't believe there is such a method established for this type of problem. Therefore, this would be an extremely high bar for a conference submission, and I explicitly refrain from asking this of the authors, but would suggest it as future work. I believe it would be a significant contribution.
    • Inference cost: Additional FLOPs and KV cache overhead during decoding would not typically affect wall-clock time for small-batch decoding, but may significantly affect large-batch decoding. Additional KV cache reads would also affect attention wall time for large-batch long-context scenarios. Shared reads from slot lanes could be optimized in attention kernels but would require significant effort. Comprehensive theoretical cost analysis and batch decoding measurements with high request loads to test the effect on concurrency would be good first steps to addressing these potential costs.
Questions for authors
  • (Minor weakness and suggestion) The methodology section is too long and the analysis (Table 2 in the discussion section) is too short. The method dives too deep into technical details and theoretical analysis while being hard to understand at the high level. A more concise high-level overview of the method with details deferred to the appendix would be more helpful to the reader. This would also allow more room to expand the analysis, which comprises of an overly compressed Table 2 which is under-explained in the accompanying section (strangely, the discussion section. The analyses are genuinely interesting and relevant to the core method but the explanation is too compressed that interpretation is nearly left to guess work. I only clearly understood about half. A dedicated analysis section with appropriate details would bring light to these results.
  • How does this method differ from Thoughtbubbles [1] and what is the unique contribution of your paper in relation to [1]? Note that [1] is only on arXiv therefore this does not affect my rating, according to ICLR 2027 guidelines.

[1] https://arxiv.org/pdf/2510.00219

10 comments
AuthorsOpenReview ↗

Responses for reviewer pkrD [1/9]

W1. Evidence for adaptive parallel computation

Overall, the evaluation and analysis is limited in many aspects and does not substantially increase confidence in the viability of pause-token style (adaptive) parallel computation, considering generality, scalability and train/inference cost tradeoffs.

A1. We thank you for your considered review.

At the reported MATH-500 threshold, persistent slots outperform the ordinary MoE control. With slots enabled, more answers change from wrong to right than from right to wrong (Tables 1–2). Imagine+ also improves on SFT on BBH Causal Judgment (Table 3), and Imagine-1 improves on SFT at 1.7B (Table 6). We report how long a single user waits between tokens (W7).

Table 1. MATH-500 accuracy by architecture and slot setting.

ConfigurationAnswer accuracy (%)
Controlled ordinary MoE40.8
Imagine+ / disabled37.0
Imagine+ / selective slots42.4

Table 2. Changes when slots are enabled for the same checkpoint. Token percentages use all teacher-forced candidate positions. Answer counts cover all 500 MATH-500 questions.

EvaluationUnitCorrectedHarmedNet
Three-slot predictions% of candidates (net: pp)24.0315.20+8.83
Imagine+ answersquestions7649+27
AuthorsOpenReview ↗

Responses for reviewer pkrD [2/9]

W2. Evaluation beyond mathematics

It is unclear if the method achieve robust performance gains due to: Limited evaluation on three math reasoning datasets on a single backbone model

A2. We thank you for your useful suggestion to broaden the evaluation.

We trained SFT and Imagine+ for five epochs on the same 19,048 OpenR1 examples filtered to 4096 tokens, excluding matches to BBH [1]. We selected the SFT checkpoint by held-out CE and chose the Imagine threshold on separate validation questions. On BBH Causal Judgment, Imagine+ with slots enabled outperformed SFT and the same checkpoint with slots disabled (Table 3). W4 reports the comparison on a second backbone size.

Table 3. BBH Causal Judgment accuracy on 187 questions. Both models trained on the same OpenR1 mixture. The controller threshold was fixed at 0.78.

Model / policyCorrectAccuracy (%)Controller gain over row (pp)
Plain SFT8847.06+11.23
Imagine+ / disabled8746.52+11.76
Imagine+ / controller10958.290.00

[1] Mirac Suzgun et al. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. Findings of ACL 2023, pp. 13003-13051. DOI: 10.18653/v1/2023.findings-acl.824. https://aclanthology.org/2023.findings-acl.824/

AuthorsOpenReview ↗

Responses for reviewer pkrD [3/9]

W3. Sensitivity to the controller threshold

Sensitivity to threshold hyperparameter: while the accuracy gains seem somewhat significant, it is highly sensitive to the threshold hyper parameter as shown in Figure 4 and 7. At many points, accuracy is lower than the SFT baseline.

A3. We thank you for your perceptive observation about threshold sensitivity.

Imagine+ matched or exceeded SFT at seven consecutive thresholds in the scan on MATH-500 (Table 4). We carried the reported threshold of 0.85 unchanged to AMC23 and OlympiadBench. Imagine+ scored above SFT on both without tuning on either benchmark (paper Table 1).

We also chose the threshold on 200 independent validation questions. The choice was again 0.85. At that threshold, enabling slots improved accuracy on AMC23 and OlympiadBench over the same checkpoint with slots disabled. Table 5 also gives the SFT results from this evaluation. AMC23 averages eight answers per question.

Table 4. Imagine+ accuracy and activation across thresholds on MATH-500.

ConfigurationAccuracy (%)Activation (%)
SFT reference40.200
Imagine+ slots disabled37.000
Reported policy, threshold 0.8542.406.37
Seven tested points, thresholds 0.80-0.8840.20-43.202.26-15.47
Near-disabled policy, threshold 0.9039.000.90

Table 5. New benchmark answers after selecting the Imagine+ threshold of 0.85 on independent validation questions. On and Off use the same checkpoint.

TaskSFT (%)Off (%)On (%)On - off (pp)On - SFT (pp)
MATH-50040.2037.0042.40+5.40+2.20
AMC2320.9421.5621.560.00+0.62
OlympiadBench13.3312.1514.81+2.67+1.48
Reviewer_VULCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IMAGINE improves difficult token predictions through selective parallel computation. It maintains slot-specific latent states across layers and uses a controller to allocate additional slots before the current forward pass. Experiments on Qwen3-0.6B show accuracy improvements across three mathematical reasoning benchmarks, while a single-slot variant offers lower decoding latency than a parameter-matched recurrent model.

Critical strengths

A concrete alternative to selective recurrence. Persistent, specialized slot states provide additional capacity without repeated sequential backbone passes. The MoE, communication, and readout ablations help assess the architectural choices (Table 2).

The evaluation considers practical responsiveness. Alongside accuracy and parameter-matched comparisons, the paper reports throughput, slow-token latency, and memory use. These measurements provide a useful picture of the deployment trade-offs (Figure 3).

Critical weaknesses

The threshold-selection protocol needs clarification. Controller thresholds are scanned on MATH-500, which is also used to report the main results. It is unclear whether the final threshold was selected independently of evaluation accuracy. The relatively small gains also lack accuracy uncertainty estimates, particularly important for AMC23’s 40 questions.

The joint accuracy–efficiency comparison is insufficiently specified. The strongest accuracy result uses Imagine+, whereas the main latency result uses Imagine-1. Although Table 1 includes Imagine-1 accuracy, activation rates and evaluation settings differ across the reported measurements. Essential details are deferred to appendices absent from the supplied PDF, making the combined trade-off difficult to verify.

Questions for authors

How was the final controller threshold selected? If MATH-500 accuracy informed the choice, do the gains persist with a threshold selected on a separate validation set?

Can you report accuracy, latency, memory, parameter count, and activation rate for the same measured configuration, together with its hardware and stopping settings?

Can you provide paired accuracy intervals and any available independent training runs? For AMC23, multiple sampled answers should be grouped by question when estimating uncertainty.

6 comments
AuthorsOpenReview ↗

Responses for reviewer VULC [1/5]

W1. Threshold selection and uncertainty

The threshold-selection protocol needs clarification. Controller thresholds are scanned on MATH-500, which is also used to report the main results. It is unclear whether the final threshold was selected independently of evaluation accuracy. The relatively small gains also lack accuracy uncertainty estimates, particularly important for AMC23’s 40 questions.

A1. We thank you for your constructive comments on evaluation.

We selected the original Imagine+ threshold on MATH-500 and carried it unchanged to AMC23 and OlympiadBench. Imagine+ scored above SFT on both target tasks without using either task to tune the controller (Table 1). Table 2 shows the original MATH-500 scan. In a second test, we chose thresholds on separate validation questions and used them to generate new benchmark answers (Q1). Q3 gives paired intervals, with the eight AMC23 answers clustered by question.

Table 1. Imagine+ accuracy on two other benchmarks with the threshold chosen on MATH-500.

BenchmarkThresholdImagine+ accuracy (%)SFT accuracy (%)Difference (pp)
AMC230.8521.5620.94+0.62
OlympiadBench0.8514.8113.33+1.48

Table 2. Historical Imagine+ scan on MATH-500.

Threshold rangeTested points matching or exceeding SFTMATH-500 accuracy range (%)
0.80-0.887/740.20-43.20
AuthorsOpenReview ↗

Responses for reviewer VULC [2/5]

W2. Joint accuracy and efficiency

The joint accuracy–efficiency comparison is insufficiently specified. The strongest accuracy result uses Imagine+, whereas the main latency result uses Imagine-1. Although Table 1 includes Imagine-1 accuracy, activation rates and evaluation settings differ across the reported measurements. Essential details are deferred to appendices absent from the supplied PDF, making the combined trade-off difficult to verify.

A2. We thank you for your helpful request to clarify the comparison.

Our most accurate model, Imagine+ at threshold 0.85, took 33.97 ms per token on average and 87.553 ms for the slowest 1% of token intervals (Section 7.2 and Table 7 of the paper). Imagine+ retains the full backbone and can activate up to three slots at a difficult token. The slots run in the same forward pass, but each adds attention and feed-forward computation. In Figure 3, the specialized Imagine-1 decoder reduced the mean latency of the slowest 1% of tokens by 34.9% relative to parameter-matched TaH. We timed both configurations on the same eight questions at batch size one. Q2, Table 4 reports each variant's MATH-500 accuracy and timing, including the repeat measurement of mean latency.

AuthorsOpenReview ↗

Responses for reviewer VULC [3/5]

Q1. Independent threshold selection

How was the final controller threshold selected? If MATH-500 accuracy informed the choice, do the gains persist with a threshold selected on a separate validation set?

A3. We thank you for your specific question about independent validation.

The original MATH-500 threshold transferred to both other benchmarks without retuning (Table 1).

On 200 held-out math questions disjoint from training and all three benchmarks (sampled from TaH's held out math set), maximizing validation accuracy and breaking ties by lower activation again selected 0.85 for Imagine+.

Reviewer_NG8mOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Imagine, a post-training method that selectively increases computational width at difficult token positions. It introduces additional hidden-state branches, called slots, which share the backbone weights but use slot-specific LoRA parameters. These states interact across Transformer layers and are merged before next-token prediction. Training proceeds in two stages: the backbone and slot modules are jointly fine-tuned using a dense-to-hard curriculum, followed by training a controller to select the number of slots based on the expected prediction benefit. Experiments on a 0.6B model across three mathematical benchmarks show modest accuracy improvements, while the single-slot variant achieves decoding throughput comparable to SFT and lower tail latency than a recurrent reasoning baseline.

Critical strengths
  1. The method offers a novel combination of persistent parallel slot states and adaptive token-level width allocation, providing an interesting alternative to recurrent depth for allocating additional computation.

  2. The paper provides detailed supplementary material and accompanying code, offering fairly comprehensive documentation of the method and experimental setup. Although I have not run the code, these resources should facilitate reproducibility.

Critical weaknesses
  1. First, I find the method fairly complex and potentially difficult to scale to larger models. My understanding is that it mainly introduces additional LoRA matrices to incorporate more hidden states at so-called hard-token positions, together with a controller that determines the number of slots, essentially judging whether a position is hard. This introduces additional trainable parameters and a two-stage training pipeline, which could lead to instability. The experiments use a 0.6B base model; with a substantially larger backbone, the additional risk of instability could be greater.

  2. The biggest issue is that the empirical gains are small, and there is no training-speed advantage. In Figure 3, only Imagine-1 achieves comparable training speed to SFT, while Table 7 suggests that Imagine+ performs substantially worse in terms of speed. Even the best-performing Imagine+ achieves an average accuracy improvement of less than 2 percentage points. Moreover, the experiments are limited to a single 0.6B model and three mathematical benchmarks, making the evaluation rather narrow.

  3. There is also a concern about tuning on the evaluation data: the method scans and selects thresholds on MATH-500, yet MATH-500 is also included in the reported evaluation results.

  4. The claim that an appropriately sized step reduces cross-entropy when the correction direction aligns with the prediction error is conditional. It does not sufficiently explain why this particular architecture and training procedure can reliably learn such corrections.

Questions for authors

See Weaknesses.

5 comments
AuthorsOpenReview ↗

Responses for reviewer NG8m [1/4]

W1. Training stability and model scale

First, I find the method fairly complex and potentially difficult to scale to larger models. My understanding is that it mainly introduces additional LoRA matrices to incorporate more hidden states at so-called hard-token positions, together with a controller that determines the number of slots, essentially judging whether a position is hard. This introduces additional trainable parameters and a two-stage training pipeline, which could lead to instability. The experiments use a 0.6B base model; with a substantially larger backbone, the additional risk of instability could be greater.

A1. We thank you for your thoughtful review.

We train the experts and backbone together, then freeze both before fitting the controller. This keeps the controller's target labels fixed. Across three independent training runs, Imagine-1 0.6B scored 41.40–42.00% on MATH-500 (Table 1). At 1.7B, Imagine-1 also scored higher than SFT under the 2048-token limit (Table 2). Each slot retains the full hidden width and adds r(din+dout)r(d_{\mathrm{in}}+d_{\mathrm{out}}) private parameters per projection.

Table 1. MATH-500 accuracy (%) across three independent training seeds for Imagine-1 0.6B (threshold 0.70, generation limit of 4096 tokens).

Training runOriginal seedAdditional seed 1Additional seed 2
Imagine-141.4042.0041.60

Table 2. MATH-500 accuracy for 1.7B models with a limit of 2048 new tokens. The TaH score is the mean across decoding seeds 42, 43 and 44.

ModelBackboneController thresholdAccuracy (%)
Plain SFT1.7Bn/a51.80
TaH1.7B0.9053.60
Imagine-11.7B0.8053.40
AuthorsOpenReview ↗

Responses for reviewer NG8m [2/4]

W2. Accuracy gains and inference cost

The biggest issue is that the empirical gains are small, and there is no training-speed advantage. In Figure 3, only Imagine-1 achieves comparable training speed to SFT, while Table 7 suggests that Imagine+ performs substantially worse in terms of speed. Even the best-performing Imagine+ achieves an average accuracy improvement of less than 2 percentage points. Moreover, the experiments are limited to a single 0.6B model and three mathematical benchmarks, making the evaluation rather narrow.

A2. We thank you for your detailed assessment of the accuracy and timing results.

Figure 3 and Table 7 of the paper measure single-user inference: how long one user waits for the next token. Imagine+ improves the original three-task average over SFT, and both parameter-matched variants also score higher than SFT (Table 3). With its specialized decoder, Imagine-1 matched SFT throughput and reduced the mean latency of the slowest 1% of tokens relative to TaH (Table 4). On all 500 questions, both Imagine variants scored above SFT. Based on the mean latency measured on the other machine, both Imagine variants decoded faster than TaH and slower than SFT (Table 6). Backbone training costs more than SFT and is comparable to TaH (Table 5).

To test reasoning beyond mathematics, we trained SFT and Imagine+ for five epochs on the same 19,048 OpenR1 examples, excluding matches to BBH [1]. We selected the SFT checkpoint by held-out CE and fixed the controller threshold before testing. On BBH Causal Judgment, Imagine+ with slots enabled outperformed SFT and the same checkpoint with slots disabled (Table 7).

Table 3. Original three-task benchmark accuracy (%).

ModelMATH-500AMC23OlympiadBenchAverageRelative gain over SFT (%)
SFT40.2020.9413.3324.820.0
TaH+39.2020.6213.9324.58-1.0
Imagine-141.4021.2514.6725.77+3.8
Imagine-340.2021.8813.7825.29+1.9
Imagine+42.4021.5614.8126.26+5.8

Table 4. Single-user timing on eight questions, each timed twice at batch size one. Imagine-1 used its specialized decoder. Tail latency is the mean of the slowest 1% of token intervals.

ModelThresholdActivation (%)Mean (ms)Tail (ms)Tail change vs TaH (%)Tokens/sMean GiBPeak GiB
SFTn/a027.9730.868-56.035.752.942.96
TaH0.901.8030.3670.0910.032.932.962.96
Imagine-10.708.2127.8945.625-34.935.863.343.34
Imagine+0.858.4333.9787.553+24.929.443.173.20

Table 5. Backbone training in eight-A100-equivalent hours, excluding oracle generation and controller fitting.

SFTImagine-1Imagine-3Imagine+TaHTaH+
2.904.955.015.095.135.09

Table 6. MATH-500 evaluation on 500 questions at batch size one. Mean latency was remeasured on another machine. Tokens/s uses that mean. Tail latency and memory are from the original evaluation with the general Imagine decoder.

ModelThresholdAccuracy (%)Effective params (M)Activation (%)Mean (ms)Worst-1% mean (ms)Tokens/sMean GiBPeak GiB
SFTn/a41.60596.050.0029.0528.99834.422.933.06
TaH0.9040.60594.952.4633.1673.14330.162.943.04
Imagine-10.7042.20594.528.3929.7681.53033.602.973.10
Imagine+0.8542.40630.806.4130.9682.28732.303.153.33

Table 7. BBH Causal Judgment accuracy on 187 questions. Both models trained on the same OpenR1 mixture. The controller threshold was fixed at 0.78.

Model / policyCorrectAccuracy (%)Controller gain over row (pp)
Plain SFT8847.06+11.23
Imagine+ / disabled8746.52+11.76
Imagine+ / controller10958.290.00

[1] Mirac Suzgun et al. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. Findings of ACL 2023, pp. 13003-13051. DOI: 10.18653/v1/2023.findings-acl.824. https://aclanthology.org/2023.findings-acl.824/

AuthorsOpenReview ↗

Responses for reviewer NG8m [3/4]

W3. Threshold selection on MATH-500

There is also a concern about tuning on the evaluation data: the method scans and selects thresholds on MATH-500, yet MATH-500 is also included in the reported evaluation results.

A3. We thank you for your important question about threshold selection.

We used MATH-500 to choose the original thresholds. The Imagine+ threshold of 0.85 transferred unchanged to AMC23 and OlympiadBench, where Imagine+ scored above SFT on both tasks (Table 3). We used answers from neither benchmark to choose the threshold.

We then tested the selection rule independently of all three benchmarks. We chose the threshold with the highest accuracy on 200 disjoint held-out questions, preferring the lower activation rate in a tie. This again gave 0.85 for Imagine+. We fixed it before generating new answers. Enabling slots improved accuracy on AMC23 and OlympiadBench and left MATH-500 unchanged. Table 9 also compares these runs with SFT. Section 6 explains both selection rules, and the paper's Table 1 identifies the selection dataset.

Table 8. Original Imagine+ threshold scan on MATH-500.

Threshold rangeTested points matching or exceeding SFTAccuracy range (%)SFT (%)
0.80-0.887/740.20-43.2040.20

Table 9. New benchmark answers after selecting the Imagine+ threshold of 0.85 on independent validation questions. On and Off use the same checkpoint.

TaskSFT (%)Off (%)On (%)On - off (pp)On - SFT (pp)
MATH-50040.2037.0042.40+5.40+2.20
AMC2320.9421.5621.560.00+0.62
OlympiadBench13.3312.1514.81+2.67+1.48

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.