acceptodds
Under review as a conference paper at ICLR 2027

Self-prompted Latent Reasoners

Abstract

Large Language Models (LLMs) reason effectively when intermediate steps are written out as chain-of-thought (CoT) text, and latent reasoning models further move the explicit, token-level processes into internal computations in latent spaces before explicit output. For frozen LLMs, latent input tuning methods train a lightweight module that inserts continuous vectors after the task question, which improves the model's task-specific understanding and stability while avoiding catastrophic forgetting. However, the vectors produced by existing methods carry mostly an adaptation to the task rather than question-specific information, as replacing them with their average over questions leaves accuracy essentially unchanged, and these methods consequently perform poorly on tasks that require composing several reasoning steps. To address this issue, we propose self-prompted latent reasoners, in which a minimal mapper transforms internal representations of the intermediate layer of a frozen LLM and feeds them back to its input embeddings. The resulting signal is input-dependent and helps the model to read its own unfinished, richer thoughts, while it requires no auxiliary model and no additional input positions. Experiments on multi-step reasoning benchmarks show that the proposed method substantially outperforms prompt tuning and existing latent reasoning methods under a matched protocol, while remaining on par with them on common arithmetic benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
latent reasoning, test-time compute, multi-hop reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_cEW2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies latent input tuning for a frozen LLM, where a small trained module produces continuous vectors that help the model reason. It first shows that for SoftCoT and EBM-CoT the injected vectors are nearly the same for every question: replacing each question's vectors by their average over the test set (or swapping them between questions) leaves CLUTRR accuracy unchanged (40.4 to 40.4, and 38.5 to 38.5), and 99.9 percent of their energy is a shared component. For LRT the loss is about 20 points. The authors then propose self prompted latent reasoners. The question is passed through the first 8 of the 28 layers of a frozen Qwen2.5 7B Instruct, the hidden state at each position goes through a trained mapper (a 3584 by 3584 matrix initialized at the identity, a bias for each round and a scalar gate, 12.9M parameters), is rescaled to the norm of the embedding table, and is added back onto the input embedding of the same token. The full model then reads the corrected question and generates the answer. Nothing is appended to the input and no auxiliary model is needed.

Results? Under a shared answer only protocol (32 new tokens), the method reaches 50.5 on CLUTRR, 88.0 on ProsQA and 99.6 on ProntoQA, above prompt tuning, SoftCoT, EBM-CoT and LRT. On GSM8K and AQuA with written solutions it is level with the trained baselines (78.7 and 61.3) and below zero shot CoT (84.5 and 80.3). Further analyses cover the choice of read layer, parameter matched comparisons, a read out probe, latency, repeated feedback rounds, depth breakdowns and a 3B host.

Critical strengths

The diagnosis of existing methods is clear, cheap to reproduce and results convincing. The average substitution and swap tests (Table 3, Appendix F) show that SoftCoT and EBM-CoT inject an essentially constant vector and that a single learned prompt does as well or better. This is a useful negative finding for anyone working on soft prompt style reasoning, and the test itself (replace per question signals by their average) is a good general sanity check that could be adopted by other papers.

Additionally, the proposed method is minimal and the supporting analyses are more careful than usual. The parameter matched comparison (rank 64 mapper reaches 76.3 on trained CLUTRR depths against 53.6 for a prompt of the same size) rules out simple capacity explanations. The read out probe comparison shows the answer is not simply decodable from the layer 8 state (probes reach at most 56.3 on trained depths against 88.8 for the method). The paper also reports its failures openly: repeated rounds with a constant read layer collapse (accuracy falls from 48.9 to 22.2 on CLUTRR, and to 0.25 percent on ProsQA from six rounds onward, where the model stops emitting a parsable answer), the unseen depths show no gain, and the arithmetic datasets show no gain.

Critical weaknesses

The main comparison is run under a protocol that favors the method, and an obvious baseline is missing. The headline multi hop results are measured with answer only supervision and a 32 token budget, which by design excludes written reasoning, so the host's own chain of thought cannot help. Where written reasoning is allowed (GSM8K and AQuA), no trained method beats the untrained zero shot CoT host, and the proposed method is slightly below prompt tuning (78.7 vs 79.6 on GSM8K). More importantly, there is no parameter efficient fine tuning baseline: a LoRA or similar adapter with a comparable parameter budget (about 12.9M) trained on the same data and loss is the natural alternative for adapting a frozen host to a task, and the related work argues against it on forgetting grounds without measuring forgetting. Without that comparison it is unclear whether question conditioning is what matters, or simply having a trained module that touches every input position. In addition, Appendix A states that the chat templates differ across method families and that the effect "has not been measured": the SoftCoT family is evaluated with a closed user turn and no assistant generation prompt, while the proposed method, prompt tuning and zero shot CoT use the host's chat template with the generation prompt. EBM-CoT also had one diverged ProsQA seed excluded from the reported mean. These are small details individually, but they all fall on the baselines' side.

Questions for authors

1/ How much does the chat template difference in Appendix A matter? Please rerun SoftCoT and EBM-CoT with the generation prompt appended, or the proposed method with their template.

2/ Were excluded runs (the EBM-CoT ProsQA seed) the only exclusions, and what is the mean including it?

3 comments
AuthorsOpenReview ↗

Response to Reviewer cEW2 (part1)

We thank the reviewer for the detailed review and questions. Here we address each of the weakness and question:

W1 on parameter efficient fine-tuning baselines:

We thank the reviewer for pointing out this interesting comparison that has not been well tested in related latent reasoning works. We tested our method with LoRA on Llama-3.1-8B across the multi-hop reasoning benchmarks.

MethodTrained paramsCLUTRRProsQAProntoQA
Prompt tuning0.52M48.8 ± 0.589.0 ± 0.899.7 ± 0.0
Ours16.8M56.2 ± 3.598.7 ± 0.399.7 ± 0.3
LoRA, rank=2017.0M57.0 ± 1.199.9 ± 0.199.7 ± 0.2

Our method remains comparable performance against LoRA with the similar amount of trained parameters. Note that our method does not touch the model’s own weights and is thus orthogonal to the fine-tuning family of methods. The more informative comparison is with the methods that keep the weights fixed, and in our measurements, other input-side tuning of a frozen model, performs below ours as shown in the main experiment. These results have been added to Appendix I, Table 13 in the revision.

On forgetting, we agree with the reviewer that weight adaptation’s forgetting was only supported through related literatures. This argument is not essential for our main contribution and has been removed.

On question conditions versus a trained model that touches every input position, we have tested in Table 5 in the paper replacing each question’s correction with the correction average over questions. The module still touches every position, yet CLUTRR accuracy drops from 50.5 to 8.4. Swapping in another question's correction gives 10.1. The gain is therefore carried by the part of the correction that is specific to the question. On the other side, LoRA does not contradict this. Its weight update is applied to each question's own activations, so its effect is also question dependent. What our method shows is that this question dependence can be obtained without touching the weights, using the frozen model's own intermediate state supplies it at the input as an orthogonal approach.

W2 on testing protocols:

Every trained method uses the same protocol: prompt tuning, SoftCoT, EBM-CoT, LRT and LoRA are all trained on answer-only targets and decoded with 32 tokens. Within the trained methods, the protocol therefore favours none of them. The untrained host's zero-shot CoT is given 2048 or 4096 tokens to write out its reasoning.

We also compare in two further settings:

  1. Written reasoning under one template (see Q1 below).
  2. Two standard benchmarks, used unmodified with their official metrics: 2WikiMultihopQA and MuSiQue (reply to Reviewer 2SKk). There, ours exceeds zero-shot CoT by 3.6 and 14.1 EM while generating 6 tokens per question instead of 158 and 209.

These details have been added to Section 4 and Section 5.2 in the revision.

Q1 on SoftCoT and EBM-CoT evaluated without the generation prompt:

We thank the reviewer for pointing out this issue. We reran the written reasoning experiments (GSM8K and AQuA) with same template for every method, including the raw question in the host chat template with the generation prompt. SoftCoT and EBM-CoT keep their soft-thought paragraph.

MethodGSM8KAQuA
Host, zero-shot CoT91.5181.10
Prompt tuning91.61 ± 0.3580.45 ± 1.64
SoftCoT91.99 ± 0.5080.58 ± 2.17
EBM-CoT91.53 ± 0.4681.76 ± 1.20
Ours91.91 ± 0.0479.66 ± 1.38

Under one template, all methods are within about one standard deviation of each other and of the host. Given that the host had been trained on much better arithmetic datasets and already have the corresponding ability, the main purpose of this experiment is to show that our method maintain the host ability to handle arithmetic tasks while improving its multi-hop reasoning performance. We have replaced Table 2 with this table and update Appendix A. These results have been updated on Table 2 in the revision.

AuthorsOpenReview ↗

Response to cEW2 (part2)

Q2 on excluding seeds:

We thank the reviewer for pointing out this ambiguity. This was the only exclusion in the original paper. We have further tested 7 seeds for EBM-CoT on ProsQA, resulting in performance of 73.62, 70.12, 2.50, 68.00, 74.25, 73.38, 70.38. On seed 44, the training diverged. Given that this is not very frequent in EBM-CoT and for a fair comparison using EBM-CoT as a baseline method, we excluded seed 44. The ranking that our method is above EBM-CoT on ProsQA remain unchanged either way, and this exclusion only favored the baseline.

For the revision we apply the same rule to every method and dataset. A seed will be excluded from the table mean if its training diverges or if its test accuracy is more than 10 points below the median of that method's seeds on that dataset. We have provided all excluded unstable seeds and their effects on the statistics in Appendix A, and these will not influence our main claims.

Reviewer_cEW2OpenReview ↗

Q1 and Q2 have been addressed

Thanks for the rebuttal, while my concerns on Q1 and Q2 have been mostly addressed, my biggest concern on LoRA remains. With LoRA at the same parameter count matching it, the method is no longer better than the obvious alternative, only different. I will hence remain my score.

Reviewer_z3EaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a simple latent reasoning method that maps intermediate activations of a frozen LLM back into its input embeddings. It trains a small mapper without additional input tokens or requiring an auxiliary model. Experiments show improvements on three synthetic reasoning benchmarks, while substitution tests and ablations examine the feedback’s question dependence.

Critical strengths
  1. Simple & practical. The method uses only 12.9M trainable parameters, without an auxiliary model or additional input tokens, making it practically appealing.
  2. Useful diagnostics. Substitution tests provide useful evidence that the question-dependent corrections matter.
Critical weaknesses
  1. Limited validation across models. The main experiments use only Qwen2.5-7B-Instruct. Appendix E includes Qwen2.5-3B-Instruct, but only on CLUTRR and within the same model family. This limited coverage leaves uncertainty about whether the benefits generalize across architectures and model scales.
  2. Arithmetic performance degrades comparing to Zero-Shot CoT. Accuracy drops by 5.8 points on GSM8K (84.5% → 78.7%) and 19.0 points on AQuA (80.3% → 61.3%). Matching other trained methods does not establish a compelling benefit when the untrained host performs substantially better. Zero-shot CoT also requires no additional parameters or feedback pass, although its longer generated reasoning means lower total inference cost cannot be assumed without a direct comparison.
Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer z3Ea

We thank the reviewer for careful reading and constructive feedback. Here we address each of the weakness and question:

W1 on more model family:

We repeated the main experiment on Llama-3.1-8B-Instruct with the same recipe. Every method used the optimal learning rate from a grid search. On Llama the method behaves as on Qwen.

MethodTrained paramsCLUTRRProsQAProntoQA
Host, direct answer014.70.846.4
Host, zero-shot CoT (2048 / 4096 tokens)033.367.166.2
Prompt tuning (128 vectors)0.52M48.8 ± 0.589.0 ± 0.899.7 ± 0.0
SoftCoT25.2M43.4 ± 0.679.4 ± 1.797.5 ± 0.5
EBM-CoT17.8M41.5 ± 1.378.8 ± 2.697.9 ± 0.4
Ours16.8M56.2 ± 3.598.7 ± 0.399.7 ± 0.3

These results have been added to Section 4, Section 5.1, and Table 3 in the revision.

W2 on arithmetic performance:

We traced the arithmetic performance gap to the training targets. On GSM8k and AQuA, every trained method learned to imitate the training set’s human reference solutions, which skips reasoning steps, uses different formats, and are much shorter (median of 146 token) compared to the host’s zero-shot solutions (median 266 tokens). Imitating this format, therefore, is forcing the host to use less explicit steps to approach the question, making them more prone to mistakes. To test this, we retrained every method on the host's own correct solutions from the training split. Note that this method is quite common among QA works, used in STaR (Zelikman et al., 2022), RFT(Yuan et al., 2023), and SDFT(Yang et al., 2024), thus a more fair comparison towards the host’s zero-shot. Each method used the same chat template with the generation prompt.

MethodGSM8KAQuA
Host, zero-shot CoT (rescored)91.5181.10
Prompt tuning91.61 ± 0.3580.45 ± 1.64
SoftCoT91.99 ± 0.5080.58 ± 2.17
EBM-CoT91.53 ± 0.4681.76 ± 1.20
Ours91.91 ± 0.0479.66 ± 1.38

Under equal terms, every method, ours included, is within about one standard deviation of the untrained host. The main purpose of this experiment is to show that our method maintains the host ability to handle arithmetic tasks while improving its multi-hop reasoning performance. We agree with the reviewer that the host has been intentionally trained on much larger and higher quality arithmetic dataset so a fine-tuning will not significantly improve its arithmetic capabilities. The method's benefit is on multi-step compositional reasoning without a written trace (Table 1, and the new 2Wiki and MuSiQue results), not on arithmetic. These results are updated on Table 2 and Section 5.1 in revision.

W2 on inference cost:

We timed every method on one A100 at batch size 1 on 50 test questions per dataset.

Table A: mean wall-clock time per question (ms), and test accuracy (%).

DatasetZero-shot CoTOursHost, direct answerAccuracy, zero-shot CoTAccuracy, ours
CLUTRR2031807315.150.5
ProsQA1435313212160.491.8
ProntoQA14155928281.799.6
GSM8K6119596091.591.9
AQuA104011077281.179.7

Table B: mean number of generated tokens per question. The first five rows are the same 50 questions as Table A; 2Wiki and MuSiQue are averaged over the full official dev sets (not timed).

DatasetZero-shot CoTOursHost, direct answer
CLUTRR9033
ProsQA62855
ProntoQA61733
GSM8K267261
AQuA465483
2WikiMultihopQA15866
MuSiQue20966

On the multi-step datasets ours is 25x, 109x and 154x faster per question than zero-shot CoT while being more accurate. It costs 9 to 12% more than the host answering directly, which is the price of the 8-layer feedback pass. These results are added in Section 6 and Appendix J in the revision.

References:

Zelikman, E., Wu, Y., Mu, J., Goodman, N. D. (2022). STaR: Bootstrapping Reasoning With Reasoning. NeurIPS 2022.

Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., Zhou, J. (2023). Scaling Relationship on Learning Mathematical Reasoning with Large Language Models. arXiv:2308.01825.

Yang, Z., Pang, T., Feng, H., Wang, H., Chen, W., Zhu, M., Liu, Q. (2024). Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning. ACL 2024.

Thanks for the extra experiments. That address my main concerns. Arithmetic is now close to zero-shot cot, though aqua is still slightly lower. I’m raising my score to 3.

Reviewer_2SKkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Self-Prompted Latent Reasoners, a method for adapting a frozen language model for latent reasoning by feeding its intermediate representations back into its input embeddings. A partial forward pass produces initial representations of the prompt, that a trainable mapper transforms and adds back to the original token embeddings. The full frozen model then processes the transformed representation and generates the answer. The method is motivated by substitution experiments showing that similar methods rely on conditioning that is task rather than query dependent. The proposed method improves accuracy on multi-step reasoning tasks. Additional analysis indicate the method is more query dependent than baselines and maintains an advantage against prompt tuning when more parameters are trained.

Critical strengths
  1. The work proposes a simple and efficient method for adapting frozen LLMs for multi-step reasoning with a negligible overhead in trained parameters for the mapper.
  2. Additional evidence suggest that the proposed method is indeed more query dependent than the baselines as intended.
Critical weaknesses
  1. It is unclear to me how reliable are reasoning methods that rely only on augmenting the query representations, without any subsequent reasoning trace. What scenarios, theory or application, are such methods intended to be applied in. Since the method does seem helpful it is interesting to check its utility on other related tasks or demonstrate a setting in which it is desired to reason without generating a trace at all.
  2. A possible limitation of reasoning only by refining the query representation, while maintaining causal representations, is that the intention is underspecified until the end of the query, implying additional compute is not be helpful.
Questions for authors
  1. Can you explain where you see such a method being used? one option is information retrieval or reducing hallucinations in queries where reasoning is not essential
2 comments
AuthorsOpenReview ↗

Response to Reviewer 2SKk

We thank the reviewer for the careful reading and constructive feedback. Here we address each of the weakness and question:

W1 and Q1 on applications:

We see the method being used in cases where a short answer is needed for multi-hop reasoning under tight token budgets. Some examples of such scenarios can be the reading step of a retrieval-augmented pipeline or an agentic step that needs only a short decision, where having a human reader or another agent to read the trace itself is not essential. Here we show results running our methods on two standard retrieval-augmented multi-hop QA benchmarks with Qwen2.5-7B-Instruct as host model:

MethodTrained params2Wiki EM2Wiki F1MuSiQue EMMuSiQue F1
Host, direct answer037.0644.2021.3131.57
Host, zero-shot CoT055.6161.7327.1837.75
SoftCoT16.5M54.66 ± 1.4361.0636.16 ± 0.5347.25
Prompt tuning (128 vectors)0.46M52.69 ± 0.4959.0036.42 ± 0.9946.71
Ours12.9M59.16 ± 0.5565.7541.32 ± 0.6152.00

These results have been added to Section 5.2 and Table 4 in the revision. More details on the experiment setups and multi-hop results are on Appendix H and Table 12.

W1 on reliability:

We agree that this is the central question for most latent reasoning methods. Theory on CoT expressivity (Feng et al., 2023; Merrill & Sabharwal, 2024) shows that without generating intermediate tokens, the computation must happen within one forward pass and that bounds the serial composition a transformer can perform. We do not claim to lift this bound on long derivations. Most latent reasoning methods therefore stand as a cost-effective alternative for CoT while keeping the same performance on less-demanding reasonings. Among these methods, ours makes the injected vector to the query more reliable for each specific question, which makes this method perform better than other latent reasoning methods that injects only a task-specific vector, as shown in Section 5.3. The related results are on Section 5.2 and 5.3 in the revision.

W2 on causal representation:

We thank the reviewer for pointing out this mechanism at an interesting angle. We agree that the correction at any prompt position is computed from the host’s layer 8 state at the same position, which under causal attention depends only on the tokens up to that position. In all datasets shown in the paper, all factual descriptions precede the final question, and thus it makes sense to question the effect of reading the factual descriptions with our method. We directly built tests to single out the effect of corrections on question and fact tokens.

Correction kept onCLUTRRProsQAProntoQA2Wiki (EM)MuSiQue (EM)
Template tokens only31.273.391.149.736.1
+ fact tokens (cannot see the question)34.189.998.955.638.8
+ question sentence38.675.394.454.138.8
All positions50.591.899.659.241.3

On every dataset, correcting all positions is better than correcting either part alone. This shows that having the model build some initial thoughts on the fact tokens is still beneficial even without knowing the question. Also given that in realistic scenarios the question’s intention may not always be at the end of a prompt, spending small extra compute on all tokens while saving subsequent CoT reasoning tokens should still have significant advantage over explicit CoT. The related results are added to Section 5.3 and Table 6.

References:

Feng, G., Zhang, B., Gu, Y., Ye, H., He, D., Wang, L. (2023). Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective. NeurIPS 2023.

Merrill, W., Sabharwal, A. (2024). The Expressive Power of Transformers with Chain of Thought. ICLR 2024.

I thank the reviewers for their thoughtful response and follow up experiments. My main concern still remains, while I agree that potentially the method may prove useful in agentic setup and lead to lower latency, that is a large claim that remains to proven via experiments. As it stands, experimental validation remains focused on multi hop QA and synthetic scope which does not support the utility of the method as part of a larger scale agentic system that requires handeling of long context and multi-turn behavior. Due to these reasons I am keeping my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Submission Revisions

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful feedback and constructive suggestions. We have carefully revised the manuscript (all changes highlighted in blue) to address their concerns. Please see below a summary of the main revisions. More details can be found in our responses to individual reviewers.

Second model family (Reviewers z3Ea, cEW2). We repeated the main experiment on Llama-3.1-8B-Instruct with every baseline, using Llama-3.2-1B-Instruct as the assistant model for SoftCoT and EBM-CoT (new Table 3, Section 5.1). The proposed method keeps the same ordering as on Qwen2.5-7B-Instruct while training fewer parameters than either assistant-based method.

Retrieval-augmented multi-hop question answering (Reviewer 2SKk). We added two standard multi-hop QA benchmarks, 2WikiMultihopQA and MuSiQue, evaluated on the full official development sets with exact match and F1 (new Section 5.2, Table 4 and Appendix H). The proposed method obtains the highest EM and F1 on both benchmarks while generating only the answer, and its accuracy stays stable as the number of hops grows.

Arithmetic under one protocol (Reviewers z3Ea, cEW2). We retrained every method on GSM8K and AQuA with the same chat template including the generation prompt and one answer reader for all methods and the zero-shot CoT anchor (revised Table 2 and Appendix A). All methods, including ours, remain at the level of the frozen LLM. We revised the Abstract, Introduction, Section 5.1 and Conclusion accordingly. The method’s benefit is on multi-step reasoning, and it maintains the host’s arithmetic capability.

Positions that carry the correction (Reviewer 2SKk). We added an analysis that keeps the correction only on the facts, which under causal attention are read before the question, or only on the question sentence (new Table 6, Section 5.3). Correcting all positions gives the highest accuracy on every dataset.

Measured inference cost (Reviewer z3Ea). We timed every method end to end on one A100 (Section 6 and new Appendix J). On the multi-step datasets the proposed method is 25 to 154 times faster than zero-shot CoT with higher accuracy. We state explicitly that there is no cost advantage on arithmetic.

Parameter-efficient fine-tuning baseline (Reviewer cEW2). We added a parameter-matched LoRA comparison (new Appendix I). We also removed the forgetting argument against weight adaptation from the Abstract, Introduction and Related Work, as it was not measured in the submission.

Evaluation protocol and seed reporting (Reviewer cEW2). We added a single seed-exclusion rule applied to every method and dataset, together with a table listing every excluded seed and the mean over all seeds (Appendix A, Table 10).

We hope these revisions can address the reviewers' concerns and make the paper clearer and stronger. We sincerely appreciate the reviewers' time and feedback, and we welcome any further questions.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.