- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors propose a method to adapt autoregressive transformers to be able to adaptively leverage parallel computation in a manner similar to pause tokens.
- Leveraging parallel computation: the model forwards the same input embeddings through the transformer in parallel "lanes" where each lane uses a unique LoRA adaptation of the backbone. The primary lane is called the real lane and the optional additional lanes are called slot lanes. A lightweight module is used to combine the outputs from multiple lanes into a single input for the shared LM head. Communication between lanes is done through causal attention between lanes. I.e., slot lanes attend to the KV cache of previous real and slot lanes within the same token.
- Hidden cost: while real lanes only attend to the KV cache of real lanes in previous tokens, slot lanes attend to not only the real lanes but also the slot lanes (with <= slot index) of previous tokens. Therefore, all slots add KV cache overhead, in addition to linear increase in FFN FLOPs and slightly super-linear cost in attention FLOPs and KV cache reads. These cost factors could be better explained in the paper -- they can be deduced but not pointed out.
- Adaptive parallelism: the method involves training a decision head (the Lookahead Gain Controller, LGC) based on the output hidden state of the previous step and the current input embedding. The decision head is supervised based on the teacher-forced cross-entropy loss at various slot budgets k.
- Curriculum: the model is adapted via SFT through two phases: (1) dense-to-hard (DHT) curriculum where the model learns to utilize slot lanes through end-to-end training with dropout-style application of slot lanes and (2) training the LGC for adaptive use of slots.
Critical strengths
- The motivation for adaptive parallel compute is clear and the methodology nails the execution, i.e., simple, comprehensive, and aligns well with the motivation, down to the details of e.g. handling inter-lane communication/attention.
Critical weaknesses
Overall, the evaluation and analysis is limited in many aspects and does not substantially increase confidence in the viability of pause-token style (adaptive) parallel computation, considering generality, scalability and train/inference cost tradeoffs.
- It is unclear if the method achieve robust performance gains due to:
- Limited evaluation on three math reasoning datasets on a single backbone model
- Sensitivity to threshold hyperparameter: while the accuracy gains seem somewhat significant, it is highly sensitive to the threshold hyper parameter as shown in Figure 4 and 7. At many points, accuracy is lower than the SFT baseline.
- It is unclear if and how this method scales across model sizes and LoRA ranks. Only a single size 0.6B and LoRA rank (16 per slot) is studied. For context to reader, hidden dim is 1024. Additional scaling studies would be reasonable, and feasible even on a relatively low compute budget. LoRA rank analysis is not critical but would be interesting.
- Cost tradeoffs are not comprehensively studied: the main cost of this method is additional FLOPs due to parallel slot lanes. As explained in the Paper Summary, each slot incurs additional FLOPs for both attention and FFN operations as well as KV cache overhead. This has implications that have not been addressed in training and inference.
- Training cost: the paper indicates 1.7x training time compared to regular SFT for the method. Additional experiments accounting for this cost, such as training-cost controlled experiments may be helpful. Note that, I believe a more principled large-scale study would be needed to obtain meaningful signal on whether the additional training cost can be justified in practice, in a similar spirit as pretraining scaling laws. I don't believe there is such a method established for this type of problem. Therefore, this would be an extremely high bar for a conference submission, and I explicitly refrain from asking this of the authors, but would suggest it as future work. I believe it would be a significant contribution.
- Inference cost: Additional FLOPs and KV cache overhead during decoding would not typically affect wall-clock time for small-batch decoding, but may significantly affect large-batch decoding. Additional KV cache reads would also affect attention wall time for large-batch long-context scenarios. Shared reads from slot lanes could be optimized in attention kernels but would require significant effort. Comprehensive theoretical cost analysis and batch decoding measurements with high request loads to test the effect on concurrency would be good first steps to addressing these potential costs.
Questions for authors
- (Minor weakness and suggestion) The methodology section is too long and the analysis (Table 2 in the discussion section) is too short. The method dives too deep into technical details and theoretical analysis while being hard to understand at the high level. A more concise high-level overview of the method with details deferred to the appendix would be more helpful to the reader. This would also allow more room to expand the analysis, which comprises of an overly compressed Table 2 which is under-explained in the accompanying section (strangely, the discussion section. The analyses are genuinely interesting and relevant to the core method but the explanation is too compressed that interpretation is nearly left to guess work. I only clearly understood about half. A dedicated analysis section with appropriate details would bring light to these results.
- How does this method differ from Thoughtbubbles [1] and what is the unique contribution of your paper in relation to [1]? Note that [1] is only on arXiv therefore this does not affect my rating, according to ICLR 2027 guidelines.
10 comments
Responses for reviewer pkrD [2/9]
W2. Evaluation beyond mathematics
It is unclear if the method achieve robust performance gains due to: Limited evaluation on three math reasoning datasets on a single backbone model
A2. We thank you for your useful suggestion to broaden the evaluation.
We trained SFT and Imagine+ for five epochs on the same 19,048 OpenR1 examples filtered to 4096 tokens, excluding matches to BBH [1]. We selected the SFT checkpoint by held-out CE and chose the Imagine threshold on separate validation questions. On BBH Causal Judgment, Imagine+ with slots enabled outperformed SFT and the same checkpoint with slots disabled (Table 3). W4 reports the comparison on a second backbone size.
Table 3. BBH Causal Judgment accuracy on 187 questions. Both models trained on the same OpenR1 mixture. The controller threshold was fixed at 0.78.
| Model / policy | Correct | Accuracy (%) | Controller gain over row (pp) |
|---|---|---|---|
| Plain SFT | 88 | 47.06 | +11.23 |
| Imagine+ / disabled | 87 | 46.52 | +11.76 |
| Imagine+ / controller | 109 | 58.29 | 0.00 |
[1] Mirac Suzgun et al. Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them. Findings of ACL 2023, pp. 13003-13051. DOI: 10.18653/v1/2023.findings-acl.824. https://aclanthology.org/2023.findings-acl.824/
Responses for reviewer pkrD [3/9]
W3. Sensitivity to the controller threshold
Sensitivity to threshold hyperparameter: while the accuracy gains seem somewhat significant, it is highly sensitive to the threshold hyper parameter as shown in Figure 4 and 7. At many points, accuracy is lower than the SFT baseline.
A3. We thank you for your perceptive observation about threshold sensitivity.
Imagine+ matched or exceeded SFT at seven consecutive thresholds in the scan on MATH-500 (Table 4). We carried the reported threshold of 0.85 unchanged to AMC23 and OlympiadBench. Imagine+ scored above SFT on both without tuning on either benchmark (paper Table 1).
We also chose the threshold on 200 independent validation questions. The choice was again 0.85. At that threshold, enabling slots improved accuracy on AMC23 and OlympiadBench over the same checkpoint with slots disabled. Table 5 also gives the SFT results from this evaluation. AMC23 averages eight answers per question.
Table 4. Imagine+ accuracy and activation across thresholds on MATH-500.
| Configuration | Accuracy (%) | Activation (%) |
|---|---|---|
| SFT reference | 40.20 | 0 |
| Imagine+ slots disabled | 37.00 | 0 |
| Reported policy, threshold 0.85 | 42.40 | 6.37 |
| Seven tested points, thresholds 0.80-0.88 | 40.20-43.20 | 2.26-15.47 |
| Near-disabled policy, threshold 0.90 | 39.00 | 0.90 |
Table 5. New benchmark answers after selecting the Imagine+ threshold of 0.85 on independent validation questions. On and Off use the same checkpoint.
| Task | SFT (%) | Off (%) | On (%) | On - off (pp) | On - SFT (pp) |
|---|---|---|---|---|---|
| MATH-500 | 40.20 | 37.00 | 42.40 | +5.40 | +2.20 |
| AMC23 | 20.94 | 21.56 | 21.56 | 0.00 | +0.62 |
| OlympiadBench | 13.33 | 12.15 | 14.81 | +2.67 | +1.48 |
Responses for reviewer pkrD [1/9]
W1. Evidence for adaptive parallel computation
A1. We thank you for your considered review.
At the reported MATH-500 threshold, persistent slots outperform the ordinary MoE control. With slots enabled, more answers change from wrong to right than from right to wrong (Tables 1–2). Imagine+ also improves on SFT on BBH Causal Judgment (Table 3), and Imagine-1 improves on SFT at 1.7B (Table 6). We report how long a single user waits between tokens (W7).
Table 1. MATH-500 accuracy by architecture and slot setting.
Table 2. Changes when slots are enabled for the same checkpoint. Token percentages use all teacher-forced candidate positions. Answer counts cover all 500 MATH-500 questions.