acceptodds
Under review as a conference paper at ICLR 2027

Write Back the : Revisiting the Same Tokens with Fresh Representations

11 views

Abstract

Transformers process information strictly forward through depth, preventing deeper computation from revisiting and refining earlier representations. To augment the standard forward pass, existing approaches either re-execute depth, incurring additional computation, or modify the residual stream using predefined directions, limiting their instance-level adaptation. Recently, inference-time feedback offers a direct mechanism for writing deeper residual states back to earlier layers, yet what should be fed back remains unclear. We argue that the depth increment , capturing newly accumulated computation between two layers, provides a more effective, composable, and scalable feedback signal than the full state. Building on this observation, we introduce ReFlux, a learnable feedback graph that dynamically selects and composes increment-carrying routes. ReFlux supports synchronous feedback to the same token and streaming feedback to subsequent tokens. Extensive experiments across various models, corpora, and benchmarks show that synchronous ReFlux consistently reduces perplexity across ten language-modeling corpora, and improves accuracy by 2.1–2.3 points, with gains reaching 4.7 points on multi-hop reasoning. Streaming ReFlux further retains most of these gains while preserving the base model's 1 theoretical backbone FLOPs. These results establish ReFlux as an efficient paradigm for unlocking the latent computational potential of LLMs, allowing them to revisit the same tokens with fresh representations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, residual stream, inference-time intervention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JiAJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReFlux, a novel framework that reuses the cross-layer representation deltas by feeding them back into shallower layers, thus unlocking the reasoning capabilities of LLMs. Through analyses of cross-layer residual representations and controlled feedback experiments, the authors identify the depth increment as a more effective and composable feedback signal than the full hidden state. Building on this observation, ReFlux introduces a learnable sparse feedback graph that dynamically selects source-target layer pairs and determines their feedback strengths based on contextual information, supporting both synchronous and streaming feedback mechanisms. Extensive experiments demonstrate improvements in language modeling and reasoning performance with tolerable inference overhead.

Critical strengths
  1. The paper is well-motivated and presents a coherent methodological development. The progression from the initial motivation through empirical analyses to the design of a more flexible and adaptive feedback mechanism is natural and logically grounded. The proposed method is intuitive to me and technically practical.

  2. The paper provides extensive experiments, covering PPL, accuracy, and efficiency, along with detailed analyses of the internal mechanisms and routing behaviors. These evaluations provide valuable evidence of the method's effectiveness and offer further insights into its underlying properties.

  3. The manuscript is well-written and clearly presented. The writing is fluent and well-organized, with a clear logical flow throughout the paper. The figures are informative and concise, effectively illustrating the key motivations, methodological designs, and experimental findings.

Critical weaknesses
  1. Potentially misleading presentation of the efficiency–quality trade-off. The paper evaluates the streaming mechanism under different execution protocols, making the reported performance and efficiency gains difficult to interpret jointly. Specifically, Tab.2 reports PPL improvements under strict full-sequence streaming, which incurs substantial sequential prefill overhead, whereas Tab.4 presents decode-only efficiency measurements alongside PPL results obtained under the strict streaming protocol. While Appendix E acknowledges this distinction and reports the substantial prefill overhead of strict streaming, the main-text presentation may give an overly optimistic impression of the practical efficiency–quality trade-off. A clearer evaluation and presentation of end-to-end efficiency and task performance under a consistent, deployment-oriented parallel-prefill streaming protocol would strengthen the practical claims.

  2. Insufficient justification for the claimed benefits in multi-hop reasoning. The paper attributes the improvements in multi-hop reasoning to alleviating the Access Bottleneck by reusing previously computed information. However, the routing decisions rely only on representations from the most recent token block, which may contain limited information about distant evidence. It is therefore unclear how such locally conditioned routing can effectively facilitate the integration of information across distant passages. The current analyses do not sufficiently substantiate the proposed explanation for the observed improvements in multi-hop reasoning.

Questions for authors
  1. I am curious about the characteristics of the delta signals that provide additional information to the original residual representations and contribute to performance improvements. Therefore, I suggest that the authors provide more detailed statistics on the magnitudes and directions of the original representations and injected deltas, such as their norm distributions and cosine similarities, to better understand how these signals complement the original representations.

  2. What explains the seemingly regular routing pattern observed in Fig.7? Both the source and target layers appear to shift almost uniformly toward deeper layers as token positions increase, while the depth gap between them remains nearly constant. Does this reflect an underlying mechanism of information propagation, or does it suggest that the router has converged to a simple, nearly deterministic routing pattern, potentially due to insufficient training? Further analysis across different inputs would help clarify whether this behavior reflects meaningful adaptive routing or a form of router collapse.

  3. Why does the learned router activate substantially fewer edges than suggested by the initial motivation? Fig.3(b) shows continued performance improvements as the number of active edges increases to 4–6, suggesting that combining more delta signals can provide complementary benefits. However, the final ReFlux model activates only around 2–3 edges on average, as reported in Tab.12. What accounts for this discrepancy? Or, could this indicate that the proposed adaptive router has not been sufficiently trained to fully exploit the benefits of combining multiple delta signals?

  4. There is a minor arrow misalignment in Fig.7(c).

Reviewer_y9aeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes using depth increments as a feedback signal in Transformers. The authors show that this approach can help LLMs recover useful capabilities by allowing later computations to reshape earlier interpretations.

Critical strengths
  • The paper addresses a timely topic that I believe is of genuine interest to the research community.
  • The results are promising and worth readers’ attention.
Critical weaknesses

My main, and only, concern is the paper’s presentation. At some point during the reading, I found myself unsure whether I was missing something fundamental about the field or whether the paper simply did not explain its ideas clearly enough. Below are several examples of issues I encountered. This list is not exhaustive.

  • The introduction does not clearly explain the distinction between previous approaches and the proposed method. The statement “Looped Transformers re-execute depth, while Recirculation returns full states to earlier layers” does not make the distinction clear to me; as written, these descriptions seem to refer to essentially the same idea. Figure 1 raises a related question: why does Recirculation require 2× FLOPs, while the proposed method requires only 1×? As I understand it, the proposed method follows a similar principle but uses depth increments rather than hidden states themselves. If the paper introduces a technique that reduces the computational cost to 1× FLOPs, could the same technique also be applied to Recirculation? If so, is this efficiency improvement an inherent property of depth increments, or is it a separate technique that could be applied to other approaches as well?
  • The explanation of the method does not resolve these questions. The following statement appears in the method section: “However, computation is directional across depth: a shallow state can be refined by later blocks, but later interpretations cannot directly revise earlier states.” I find this difficult to reconcile with my understanding of the proposed method. Suppose we have two layers, (s) and (d), where (d > s) and (d) is closer to the LM head. If the representation (z_d(t)) can be used to modify (z_s(t)), then the subsequent computation of (z_s(t+1)) should incorporate information from the deeper layer. Under this interpretation, I do not understand why Recirculation requires 2× FLOPs. If this interpretation is incorrect, I would expect the paper to explain why the proposed method requires only 1× FLOPs while Recirculation cannot achieve the same efficiency, and to clarify how the proposed method works in practice.
  • The subsequent sections do not address these fundamental questions, and the figures do little to clarify the method. After this point, I tried to continue reading in the hope that the remaining sections would resolve my confusion, but I could not find an explanation that addressed the issues above. Figure 4 is a particularly clear example: it presents a complex architecture and execution schedule, with numerous arrows and interacting components, yet the caption provides only a generic description: “Overview of ReFlux Architecture and Execution Schedules.” I believe the authors should guide readers through the figure and explain the key mechanisms, rather than leave readers to reconstruct the method themselves. At this point, I moved on to the experiments.
  • The experimental section does not adequately explain the implications of the results. The experiments begin on page 7, leaving relatively little room for a thorough discussion of the findings. The paper presents a number of tables and figures, but I struggled to understand what conclusions readers should draw from them and how the results support the paper’s central claims.

Overall assessment

I believe the paper would benefit from a substantial rewrite, potentially from scratch. The underlying idea appears interesting, and the results deserve attention, but the current presentation makes it unnecessarily difficult to understand the proposed method, distinguish it from prior work, and interpret the experimental findings. In its current form, I was unable to develop a clear understanding of the paper’s contribution.

Questions for authors

See weaknesses

Reviewer_4tHFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ReFlux, a feedback mechanism that feeds the hidden states in deeper layer back to the shallower layer to augment the standard forward computation. It trains a lightweight router to decide the source layer and target layer, routing the depth increment Delta=zs−zdDelta=z_s-z_d to shallow layers, so that shallow layers can know more information. Synchronous execution applies reference increments to the same token in a second traversal, while streaming extracts increments from the preceding token into the current traversal. The paper trains the router in synchronous way, implement the router to both synchronous and streaming way, and reports 2.1-2.3 points accuracy improvement, with streaming retains much of the benefit (of synchronous router) at low decoding overhead.

Critical strengths
  1. The proposed method is efficient. Streaming ReFlux improves downstream quality with only 1.02–1.08x baseline decoding runtime.
  2. The router design is well-structured. Separate gate and amplitude heads control edge selection and incremental strength, while block-level decisions reduces routing cost. Training only this lightweight controller over a frozen backbone provides an economical way to reuse depth increments.
Critical weaknesses
  1. The reported accuracy gain is marginal compared to the baseline. As shown in Table 2, on the first five benchmarks, ReFlux-streaming improves over the TF-Loop baseline by less than 1% in accuracy. Moreover, on MMLU, ReFlux-streaming even underperforms TF-Loop. These results raise concerns about whether the proposed method provides a consistent and practically meaningful performance improvement over the baseline.
  2. The explanation of transfer from synchronous training to streaming inference is insufficient. The authors suggest that earlier deep computation can serve as a predictive signal for subsequent tokens (§4.3, Figure 5, and §5.3). However, they provide limited direct evidence for the explanation, although the results show improvement.
Questions for authors
  1. What are the results when a streaming router is trained and then evaluated under both synchronous and streaming inference settings?
  2. What is adjacent-token increment directions for fixed source–target layer pairs at different positions? Please examine and relate their consistency to streaming loss improvement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.