acceptodds
Under review as a conference paper at ICLR 2027

Boundary-Resolved Answer Relays in Latent Reasoning

Abstract

How does answer information pass from continuous latent reasoning to discrete generation? We test whether the contextual cache of a naturally generated answer prefix relays information from native latent states. We introduce Boundary-Resolved Answer Transport and Sensitivity, a protocol that crosses latent and prefix caches, recomputes the terminal reader, and measures donor-answer transfer separately from destructive sensitivity. In CODI-GPT-2, native latent values repeatedly write answer information into the contextual values of “The answer is,” which terminal attention heads combine to produce an answer. On 64 new arithmetic pairs, the fixed setup yields 61 donor-answer transfers and 64 recipient recoveries under unrestricted greedy generation. All three prefix positions are required by the native-patching criterion, but redistributing their mean value difference at matched readout dose enables single-position control. In two prospectively selected Llama-based checkpoints, the same bidirectional test supports relay sufficiency in SIM-CoT, whereas Zen-E falls below the sufficiency threshold. A separate fresh-input study shows that Zen-E's patched prefix can recover the original donor answer with latent support from different-answer problems, although latent content still influences the output. Complementary boundary studies show persistent latent-answer transport and transport–sensitivity dissociation. Together, these findings support an operator-defined latent-to-prefix relay and show why answer-transport claims must specify the intervention, upstream state, and decoding boundary.

Primary area:
interpretability and explainable AI
Keywords:
latent reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JUecOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how answer information is transmitted from continuous latent reasoning states to answer generation via introducing BATS, a causal intervention protocol that crosses latent and generated-prefix cache states. Experiments on CODI–GPT-2 support a latent-to-prefix relay in which latent values repeatedly write answer information into contextual prefix states, which are subsequently read by terminal attention heads. Overall, the results characterize how answer transport depends on the intervention, upstream latent states, and decoding boundary

Critical strengths
  1. The paper addresses a concrete question about how answer information is transferred from latent states to the generated prefix
  2. The experiments include several useful controls, particularly the separation between donor-answer transfer and destructive sensitivity. The authors also validate the localized mechanism on 64 new examples with fixed intervention settings.
  3. The comparison across different latent reasoning models is useful
Critical weaknesses
  1. confusion about core claim:

Mechanistically, a prefix token's contextual K/V is by construction a function of the full prior computation, so (I suppose that) it's unsurprising that the KV cache of "The answer is" already encodes the answer and is read out downstream.

  1. Does Information Transport Constitute a Reasoning Mechanism?

A previous study (https://arxiv.org/pdf/2606.12689) found that certain patterns previously interpreted as evidence of latent reasoning also emerge in control models without the corresponding recurrence or training curriculum. So it raises my concern:

Is the observed latent-to-prefix relay specific to latent reasoning, or is it simply a more general information-transfer mechanism commonly found in Transformers?

Questions for authors

While the causal intervention protocol is carefully designed, the broader significance of the identified relay remains unclear.

In particular, the experiments establish that answer information can be transferred through contextual prefix states, but can you consider providing limited evidence of how this mechanism relates to the model's actual reasoning process or downstream reasoning performance?

Reviewer_yY3JOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how information from continuous latent reasoning reaches the final answer during token generation. The authors introduce BATS, a causal intervention protocol that crosses latent and generated-prefix KV caches to test whether the prefix can relay answer information. The protocol measures donor-answer transfer and recipient recovery separately from destructive sensitivity.

On CODI–GPT-2, the authors identify a set of attention heads that read answer information from contextual prefix values. The mechanism is confirmed on 64 new arithmetic pairs, achieving 61/64 donor transfers and 64/64 recipient recoveries. The paper further shows that the apparent need for three prefix positions depends on the intervention: redistributing their combined value difference allows a single position to control the answer.

Experiments on SIM-CoT and Zen-E reveal different prefix-sufficiency patterns. Additional controls show that useful latent support does not always require the same final answer as the original donor. The authors also examine the distinction between answer transport and destructive sensitivity across decoding boundaries.

Critical strengths
  • Clear causal evidence for a latent-to-prefix answer relay. The four-condition design is a useful extension of standard cache interchange experiments. The forward and reverse interventions help distinguish a prefix that carries donor-specific information from a change that merely disrupts the recipient's answer. I particularly appreciate the fixed-mechanism confirmation on 64 new pairs, where the same reader heads and intervention rules achieve 61/64 donor transfers and 64/64 recipient recoveries. The writer factorization and wrong-location controls provide additional evidence for the proposed information flow.

  • Careful treatment of intervention-dependent conclusions. The paper does not equate successful patching with natural necessity. The comparison between native-value and matched-dose interventions is particularly informative, since it shows that the apparent minimal prefix coalition depends on how the intervention is constructed. I also appreciate the checkpoint comparison and the separate treatment of transport and destructive sensitivity. The authors explicitly distinguish supported results from inconclusive or non-estimable cases.

Critical weaknesses
  • Limited generalization of the identified mechanism. The detailed writer–prefix–reader mechanism is confirmed primarily on one CODI–GPT-2 checkpoint using templated arithmetic problems and a fixed answer prefix. The Llama-based experiments provide whole-segment relay evidence but do not replicate the same head-level mechanism. The results therefore establish a local mechanism rather than a general account of latent-to-token information transfer across models and reasoning tasks.

  • The mechanistic interpretation remains dependent on the intervention. The main localization uses selected reader heads and cross-cache interventions that construct hybrid internal states. The three-position minimum disappears when the pooled value difference is redistributed at matched dose. These results establish that the identified prefix states can control the answer under the tested operators, but do not establish that the same path is necessary or unique during unmodified generation. The transport–sensitivity comparison is also limited by unequal segment widths and non-estimable later-boundary retention.

Questions for authors
  • Can the authors further distinguish prefix relay sufficiency from natural necessity? In particular, is there evidence that selectively disrupting the identified prefix-to-reader path during an otherwise unmodified trajectory changes the answer, while preserving alternative routes from earlier latent states?

  • How much of the three-position effect is explained by intervention strength rather than the distribution of information across positions? The matched-dose experiment changes both the injected content and its scaling. A more controlled comparison could clarify whether the native coalition reflects distributed encoding or simply the amount of signal received by the terminal reader.

  • How should the difference between SIM-CoT and Zen-E be interpreted mechanistically? The results establish different operational sufficiency outcomes, but the crossed states may also differ in compatibility with the original latent background. Is there evidence that the difference reflects distinct information-transfer mechanisms rather than stronger interactions between latent and prefix states?

  • How closely does BATS generalize beyond the fixed arithmetic templates and short answer prefixes? Evidence from a different reasoning task or a naturally varying prefix length would help establish whether the relay is a common mechanism or specific to the tested generation format.

Reviewer_7LXmOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how information passes from hidden reasoning states to the final answer. The authors introduce BATS, a method that alters the information in hidden states and answer prefixes. Their results show that information can pass within the answer prefixes, although this depends on the model and intervention.

Critical strengths

Interesting problem: Understanding how information passes from hidden reasoning states to the final answer is an important question for language models. This paper investigates that question using a method that looks at both passing and recovering answers. Well designed experiments: After setting the intervention rules, the authors tested on new examples and included many controls. For the CODI–GPT-2 model, they achieved 61/64 successful answer passes. This provides strong evidence that the method achieves the desired mechanism.

Critical weaknesses

Not generalizable: Small models and simple math tasks produced the best results. Whether the same results would hold for larger models and more realistic reasoning tasks is not shown in the paper.

Dependent on interventions: How the hidden states are altered affects the results. With the original intervention, the method requires 3 different prefixes, whereas only 1 is needed when information is redistributed. This makes it unclear how much the experiments tell us about the model’s natural behavior.

Not very novel or insightful: Previous work such as SCIT, which already looks at transporting answers using hidden state interventions, forms much of the basis for this paper. BATS provides more detailed testing, but it is not fully clear how much new contribution the results offer. The different behaviors between models are also not fully explained.

Questions for authors

Could the authors test this method on larger models and more complex reasoning tasks? This would help establish whether the results are generalizable.

How can we be sure that the identified prefixes are naturally important for generating the answer? Looking more closely at the effect of content versus intervention would help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.