- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ORION, which supervises intermediate Transformer representations using simulator-derived execution states. The authors study its interaction with positional encoding through a 2×2 comparison of absolute positional embeddings versus RoPE and auxiliary state supervision versus no supervision. Experiments cover Folding, GridShift, and an eight-algorithm CLRS pilot, with additional scaling, transfer, and representation analyses. The results suggest that state supervision can improve depth extrapolation when combined with RoPE, particularly at the largest model size. However, the evidence does not establish the stronger claims concerning orthogonal failure modes, the necessity of canonical alignment, or confinement to a true execution manifold.
Critical strengths
-
The question of how positional encoding and intermediate supervision interact in algorithmic generalization is relevant. The main factorial comparison and the two distinct synthetic domains provide a useful starting point.
-
The 26M-parameter result is promising: ORION + RoPE significantly improves Folding depth extrapolation, with positive improvements in all five seeds.
-
The cross-domain pretraining results suggest potential transfer benefits. The explicit reporting of negative mechanistic findings is useful, including unsuccessful causal interventions and the failure of the original compression hypothesis.
Critical weaknesses
-
The paper's objective and its methodological contribution are insufficiently specified. ORION needs a clearer account of what its particular representation and objective add, supported by a comparison under the same RoPE backbone and supervision budget.
-
The representation analyses do not establish the proposed confinement mechanism. Figure 4 measures displacement in a two-dimensional PCA projection, rather than distance to valid execution states or violations of the true transition. Figure 1 also lacks a RoPE-only readout comparison.
-
The negative controls do not isolate canonical alignment. Section 5.3 interprets the failure of N1–N5 as evidence that exact canonical alignment is necessary. However, Appendix D.3 states that all five controls use absolute positional embeddings, whereas the full model uses RoPE. Since Table 2 identifies RoPE as the dominant contributor to extrapolation, this comparison confounds the supervision target with positional encoding.
Questions for authors
-
How are token length, state size, and execution depth independently controlled? What changes when the size split avoids an untrained size-token embedding?
-
What is the complete implemented objective, and which component accounts for the improvement over ordinary intermediate-state prediction?
Response to the Authors' Rebuttal
I apologize for the somewhat delayed response. Thank you for the response. I understood your explanations, but concerns about the contribution of this study remain, and some parts of the response have not been reflected in the manuscript.
[Q1] Differentiation from existing intermediate supervision
I understood the explanation that supervision is applied to predict intermediate states from internal representations. However, this can also be viewed as a standard auxiliary-task formulation, so ORION's specific contribution remains unclear. Without comparisons with methods such as scratchpads that use the same intermediate-state information, it is difficult to distinguish the effect of additional ground-truth information from the effect of ORION's specific design.
[W1,Q2] Application to other backbones
From Equations (2) and (4), ORION appears applicable to other backbones in a plug-in manner. Therefore, I would like to see comparisons with and without ORION on the same backbone. It is also disappointing that there are no experiments with the Mamba family, which has addressed the problems of existing SSMs.
“Applied only to Transformers” and “designed to address the limitations of Transformers” are different. From the very first sentence of the Abstract, this paper frames the problem around the latter, but it lacks a Transformer-specific design rationale or comparisons on other backbones. Simply limiting the experimental scope to Transformers does not resolve the concerns about the contribution.
[W2,Q3] Orthogonality and interpretation of the performance figures
I also noted your response that functional orthogonality has not been established and that ORION provides an additional benefit alongside RoPE's dominant contribution. However, the following sentence remains in the abstract.
The figures of 66.7% and 59.8% are the depth-extrapolation performance of RoPE+ORION on Folding and GridShift, respectively. I still do not understand why these are described separately as RoPE's length-extrapolation performance and ORION's depth-extrapolation performance, and this needs to be corrected.
[W2,Q4] Necessity of canonical alignment
I understood your response that the current controls do not establish necessity and that you withdraw this claim. However, the necessity claim remains in the title and text of §5.3, so this needs to be revised. Furthermore, it remains unverified what advantages the canonical representation offers over other information-preserving representations.
[Q5] Generalization settings and central contribution
Thank you for explaining the experimental settings in detail. The explanation that token length and execution depth are not independently controlled should also be reflected in the manuscript.
If the contribution of this study is an analysis showing that “even exact intermediate-state supervision does not guarantee extrapolation,” this does not seem consistent with the current flow of the paper. The manuscript focuses on demonstrating ORION's superiority, but the novelty of ORION itself is not sufficiently clear either. The organization and claims of the paper need to be aligned with its central contribution.
[Q6] Related work and bibliographic information
I confirmed that the bibliographic information I pointed out has been corrected. Thank you.
Overall, I understood the response, but the contribution that this study can claim appears to be substantially reduced as a result. Furthermore, in the current PDF, it is difficult to identify revisions or additional experiments addressing the other questions beyond the bibliographic corrections in Q6. In particular, the concerns about the specific contribution over existing intermediate supervision and the lack of comparative experiments remain unresolved, so I would like to maintain my original score.