- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ReFlux, a novel framework that reuses the cross-layer representation deltas by feeding them back into shallower layers, thus unlocking the reasoning capabilities of LLMs. Through analyses of cross-layer residual representations and controlled feedback experiments, the authors identify the depth increment as a more effective and composable feedback signal than the full hidden state. Building on this observation, ReFlux introduces a learnable sparse feedback graph that dynamically selects source-target layer pairs and determines their feedback strengths based on contextual information, supporting both synchronous and streaming feedback mechanisms. Extensive experiments demonstrate improvements in language modeling and reasoning performance with tolerable inference overhead.
Critical strengths
-
The paper is well-motivated and presents a coherent methodological development. The progression from the initial motivation through empirical analyses to the design of a more flexible and adaptive feedback mechanism is natural and logically grounded. The proposed method is intuitive to me and technically practical.
-
The paper provides extensive experiments, covering PPL, accuracy, and efficiency, along with detailed analyses of the internal mechanisms and routing behaviors. These evaluations provide valuable evidence of the method's effectiveness and offer further insights into its underlying properties.
-
The manuscript is well-written and clearly presented. The writing is fluent and well-organized, with a clear logical flow throughout the paper. The figures are informative and concise, effectively illustrating the key motivations, methodological designs, and experimental findings.
Critical weaknesses
-
Potentially misleading presentation of the efficiency–quality trade-off. The paper evaluates the streaming mechanism under different execution protocols, making the reported performance and efficiency gains difficult to interpret jointly. Specifically, Tab.2 reports PPL improvements under strict full-sequence streaming, which incurs substantial sequential prefill overhead, whereas Tab.4 presents decode-only efficiency measurements alongside PPL results obtained under the strict streaming protocol. While Appendix E acknowledges this distinction and reports the substantial prefill overhead of strict streaming, the main-text presentation may give an overly optimistic impression of the practical efficiency–quality trade-off. A clearer evaluation and presentation of end-to-end efficiency and task performance under a consistent, deployment-oriented parallel-prefill streaming protocol would strengthen the practical claims.
-
Insufficient justification for the claimed benefits in multi-hop reasoning. The paper attributes the improvements in multi-hop reasoning to alleviating the Access Bottleneck by reusing previously computed information. However, the routing decisions rely only on representations from the most recent token block, which may contain limited information about distant evidence. It is therefore unclear how such locally conditioned routing can effectively facilitate the integration of information across distant passages. The current analyses do not sufficiently substantiate the proposed explanation for the observed improvements in multi-hop reasoning.
Questions for authors
-
I am curious about the characteristics of the delta signals that provide additional information to the original residual representations and contribute to performance improvements. Therefore, I suggest that the authors provide more detailed statistics on the magnitudes and directions of the original representations and injected deltas, such as their norm distributions and cosine similarities, to better understand how these signals complement the original representations.
-
What explains the seemingly regular routing pattern observed in Fig.7? Both the source and target layers appear to shift almost uniformly toward deeper layers as token positions increase, while the depth gap between them remains nearly constant. Does this reflect an underlying mechanism of information propagation, or does it suggest that the router has converged to a simple, nearly deterministic routing pattern, potentially due to insufficient training? Further analysis across different inputs would help clarify whether this behavior reflects meaningful adaptive routing or a form of router collapse.
-
Why does the learned router activate substantially fewer edges than suggested by the initial motivation? Fig.3(b) shows continued performance improvements as the number of active edges increases to 4–6, suggesting that combining more delta signals can provide complementary benefits. However, the final ReFlux model activates only around 2–3 edges on average, as reported in Tab.12. What accounts for this discrepancy? Or, could this indicate that the proposed adaptive router has not been sufficiently trained to fully exploit the benefits of combining multiple delta signals?
-
There is a minor arrow misalignment in Fig.7(c).