acceptodds
Under review as a conference paper at ICLR 2027

Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus

Abstract

Hybrid linear attention large language models (HLA LLMs) interleave linear and full attention to balance efficiency and capacity, yet how hybridization reshapes internal activation dynamics remains poorly understood. We present the first systematic study of massive activations (MAs) in layer-interleaved HLA LLMs, examining their architectural organization, training-time emergence, underlying mechanisms, and functional significance. Across five linear attention architectures, six hybridization configurations, and five input domains, we identify two architecture-aligned morphologies: pre-attention spikes (PAS) immediately before full attention and inter-spike plateaus (ISP) persisting through intervening linear attention layers. Denser full attention increasingly connects PAS through ISP, approaching the persistent MAs of conventional Transformers. This organization also recurs across 12 public checkpoints spanning 1.2B–397B parameters, covering linear attention and state-space hybrids. Controlled pretraining of Gated DeltaNet (GDN) hybrids up to 1.3B reveals early emergence and consolidation of both morphologies, alongside asymmetric gating effects. Specifically, full attention output gates strongly attenuate MA magnitudes without eliminating their organization, whereas removing GDN output gates yields modest amplification. Mechanistically, we develop a shared systematic-outlier account: PAS follows a localized write–sink–cancel process, while ISP is consistent with delayed cancellation. Functionally, our interventions show that deleting only the four largest-magnitude PAS coordinates at each full attention input reduces mean downstream accuracy by 21.9%–63.6% relative to normal inference. Moreover, reference-conditioned spike-to-plateau connection consistently improves mean real-world retrieval accuracy, yielding relative gains of 1.1%–12.6% without retraining. Anonymized code is provided in the supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hybrid linear attention, massive activations, attention sinks, large language models, model interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_A5hkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper has explored the massive activations in HLA LLMs. There are two activation patterns related to the full-attention layer placement, i.e., PAS and ISP. The training analysis shows that PAS and ISP emerge early in pre-training, and the output gating reduces their magnitude but does not eliminate their layer-wise structure. Furthermore, the paper also introduces a mechanism where the model writes anomalous activations that turn the corresponding token into an attention sink. Experiments show that removing the four largest PAS coordinates from each full attention input significantly degrades average task accuracy.

Critical strengths
  • This paper reveals two architectural coordinates in HLA LLMs, i.e., PAS and ISP. These two coordinates are related to the massive activations in HLA LLMs.

  • The experiments cover five linear attention architectures, and the observations are conducted based on open-source models including Kimi Linear, Qwen, Nemotron, and Zamba.

  • The paper presents an interesting observation, i.e., removing a few PAS coordinates is harmful to the performance of the LLM. The observations provide useful evidence for the functional significance of these coordinates.

Critical weaknesses
  • The write-sink-cancel mechanism is primarily validated using fixed coordinates within a representative GDN model. The latency cancellation observed in ISPs is largely inferred from the cross-layer phenomena. Because the maximum feature is re-selected at each layer in some analyses, it cannot be confirmed that the exact same information or coordinate is being propagated throughout.

  • Although the study covers a range of models from 1.2B to 397B parameters, the primary evidence consists of a small number of representative inputs and trajectory plots for the first token. It lacks the comprehensive statistics, confidence intervals, and token-level comparisons found in the M-A-P model suite.

  • The experiments lacked sufficient controls, such as magnitude-matched non-PAS coordinates, other tokens, or random coordinates. As a result, the performance drop might stem partly from the removal of extreme coordinates themselves instead of being specific to the PAS structure.

  • The proposed method requires an initial standard forward pass to obtain a reference trajectory and save intermediate activations. However, since it simultaneously modifies these activations and restores endpoints, it is difficult to pinpoint exactly which operation yields the benefits. Given the marginal overall improvement and performance degradation observed in some tasks, it cannot yet be considered a retrieval-augmented method ready for direct deployment.

Questions for authors

Please refer to the Critical Weaknesses part. I am glad to raise my score if my concerns are well addressed.

3 comments
AuthorsOpenReview ↗

Response to Reviewer A5hk (Part 1/2)

We thank the reviewer for the constructive questions, which help us strengthen the evidence for PAS, ISP, and the proposed write–sink–cancel process in hybrid models.

Q1. Does ISP follow the same coordinates across layers?

To distinguish coordinate persistence from a changing maximum, we track sink coordinates in the M-A-P GDN 340M and 1.3B hybrids (both 3:1). Top-1 identity measures whether adjacent residual boundaries share the largest absolute coordinate; top-4 overlap measures the fraction shared among their four largest coordinates. Anchor membership measures how often the left FA input's top coordinate remains among the top four later in the interval. The anchor is fixed once per interval, even when the maximum shifts. The natural inputs exhibit a common leading sink trajectory in these two models.

ModelTop-1 identityTop-4 overlapAnchor in top 4
340M73.91%90.22%100.00%
1.3B86.96%86.96%90.48%

The top-1 coordinate is usually preserved, with high top-four overlap and anchor membership. These results indicate that ISP typically involves a persistent dominant coordinate or a stable set of dominant coordinates, rather than merely a changing maximum. This is consistent with our fixed-coordinate PAS observations in Appendix E.1 (Figs. 13–15). We add the ISP analysis there, providing further support for the write–sink–cancel account.

Q2. Do PAS and ISP generalize across inputs and tokens in the 12 large models?

Due to space constraints, the submission primarily presented representative activation trajectories rather than comprehensive statistics for all 12 models. We now quantify PAS and ISP across five natural domains, comparing attention-sink tokens with two non-sink tokens at early and middle positions, both outside the attention top three. SSA measures pre-FA peak alignment; peak excess measures local spike prominence. ISR measures retention relative to the weaker endpoint, while absolute height measures raw plateau magnitude. The table reports five-domain averages for Qwen397B, with 95% source-cluster intervals in brackets.

TokenSSA (%)Peak excessISRAbsolute height
Sink99.21 [99.02, 99.40]0.970.7423.75 [23.61, 23.89]
Early non-sink67.83 [67.19, 68.34]0.280.885.69 [5.61, 5.74]
Middle non-sink65.99 [65.48, 66.51]0.230.905.69 [5.63, 5.74]

Sink tokens exhibit pronounced PAS and ISP patterns across the 12 models, with absolute plateau heights exceeding both controls in every model. Non-sink trajectories generally show weaker spikes; their low, relatively flat activation curves can nevertheless yield high relative ISR despite low absolute height. These findings are consistent with the token-level observations in M-A-P. §3.4 and Appendix C.3 provide the full comparisons and intervals for all 12 models.

AuthorsOpenReview ↗

Response to Reviewer A5hk (Part 2/2)

We continue with the suppression controls and connection ablations.

Q3. Do magnitude-matched or random non-PAS deletions produce the same loss?

We examine whether the observed degradation is specific to PAS coordinates. In the same M-A-P hybrids, even the four largest eligible non-PAS coordinates fail to match the PAS deletion L2 magnitude within ±5% (0/4,096 FA events per model). Thus, a magnitude-matched non-PAS four-coordinate set is unavailable at the tested sites. We compare zeroing the first token's four largest-magnitude coordinates at each FA input, replacing them with their channel means over the unedited prompt, and zeroing four random coordinates outside the local top four. Recall is mean accuracy over FDA, SWDE, Based SQuAD, and Based NQ; commonsense is mean accuracy over PIQA and ARC-Easy (equal task weights). PPL is WikiText-103 continuation perplexity (lower is better).

Model / metricNormalTop4 zeroMean fillRandom4 zero
340M Recall mean (%)28.914.4921.8829.30
340M Commonsense mean (%)62.8558.3359.0562.72
340M PPL14.6151.9719.9614.61
1.3B Recall mean (%)47.0728.9121.8847.07
1.3B Commonsense mean (%)72.8057.9265.2272.80
1.3B PPL9.5014.7314.889.50

Mean replacement also causes substantial degradation without zeroing: recall decreases by 7.03/25.20 points, commonsense accuracy drops, and PPL increases. Random-four deletion, by contrast, remains close to normal performance. These results provide additional evidence for the importance of PAS, as discussed in §4.3 and Appendices F.4–F.6.

Q4. Which part of connection improves retrieval?

We separately evaluate reference-floor editing between FA inputs, endpoint restoration, and their combination. The metrics are as described above.

Model / metricNormalInterior onlyEndpoints onlyCombined
340M Recall mean (%)28.9133.9828.9133.98
340M Commonsense mean (%)62.8562.7062.8563.00
340M PPL14.6114.6014.6114.60
1.3B Recall mean (%)47.0750.5947.0749.80
1.3B Commonsense mean (%)72.8072.7272.8072.82
1.3B PPL9.509.499.509.50

Interior editing provides the recall gains (approximately +5.1/+3.5 percentage points), with little change in commonsense accuracy and PPL. Endpoint restoration alone reproduces normal performance, while adding restoration provides no further mean recall benefit. We therefore focus on interior-only editing in §4.3 and Appendices F.1/F.4. Extending these benefits to broader tasks and settings, while further optimizing activation editing, remains an important direction toward practical deployment.

Thank you for the detailed response. The new cross-model and token-level statistics address my concern about the generality of PAS and ISP, and the ablation clarifies which edits drive the average retrieval gains in the two GDN models tested. Therefore, I have decided to raise my score to 3.

In a future revision, I encourage the authors to strengthen the evidence for delayed cancellation along fixed ISP coordinates across architectures, better isolate PAS-specific effects from perturbation magnitude, and report end-to-end inference costs including the reference forward pass.

Reviewer_Y9eMOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents an empirical study of massive activations—hidden-state entries with unusually large absolute values—in hybrid linear attention (HLA) LLMs. The authors identify pre-attention spikes (PAS) immediately before full attention and inter-spike plateaus (ISP) across intervening linear attention layers. The study combines controlled pretraining, analyses of public checkpoints, and functional interventions.

Critical strengths
  1. The paper is clearly written and easy to follow.
  2. The experimental coverage is extensive, spanning multiple architectures, hybridization configurations, input domains, and model scales, including public checkpoints up to 397B parameters.
  3. The topic is timely given the increasing adoption of hybrid attention architectures.
  4. The reference-conditioned connection intervention provides encouraging evidence of functional relevance. Mean retrieval accuracy improves across all eighteen evaluated GDN model–budget settings, and the authors report paired uncertainty estimates and retain individual task regressions.
Critical weaknesses
  1. The computational significance of PAS and ISP remains insufficiently explained. The paper convincingly establishes recurring architecture-aligned patterns, but it is less clear what these patterns reveal about how linear and full attention cooperate. What computational role do the spikes serve, and why should their persistence improve retrieval?
  2. The deletion experiment does not isolate the importance of PAS morphology. Removing dominant coordinates establishes that the trained models depend on those coordinates, but does not distinguish the importance of their magnitude from that of their pre-attention localization. Furthermore, deletion occurs before normalization and can change the entire normalized representation. Comparisons with perturbations matched in magnitude at other tokens and layer positions would strengthen the interpretation.
  3. The mechanistic account remains largely descriptive. The write–sink–cancel account characterizes the observed activation lifecycle, but does not fully explain why the interaction between linear and full attention produces it. A mathematical explanation or causal experiments testing explicit predictions would strengthen this contribution.
  4. The connection results support a narrower conclusion than their broad presentation may suggest. The reported 1.1%–12.6% relative gains concern the mean over four retrieval benchmarks in six 340M/1.3B GDN checkpoints. These results are promising, but do not establish comparable gains across the broader architectures and scales studied elsewhere. Moreover, the intervention combines intermediate activation edits with reference endpoint restoration, so the contribution of extended persistence is not isolated. The additional reference pass and trajectory storage also leave its practical accuracy–cost tradeoff unresolved.
Questions for authors
  1. What computational roles do PAS and ISP serve, particularly in retrieval?
  2. Could a simple intervention such as SoftCap, as used in Kimi-K3, mitigate massive activations? How would this affect PAS, ISP, and downstream performance?
  3. To what extent do PAS and ISP arise from the interaction between linear and full attention? Could you provide a simple theoretical account or an intuitive explanation?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Y9eM (Part 1/2)

We thank the reviewer for the constructive feedback. We clarify our contributions and provide additional experiments on PAS suppression, soft-cap training, and connection.

Q1. What are the roles of PAS and ISP, especially in retrieval?

Our work characterizes PAS and ISP in hybrid models, investigates their write–sink–cancel process, and examines their functional relevance through targeted suppression and persistence interventions.

On the M-A-P GDN 340M/1.3B hybrids (both 3:1), suppressing PAS reduces mean recall from 28.91%/47.07% to 4.49%/28.91%, while connecting the spikes into an ISP through interior editing increases recall to 33.98%/50.59%. Recall averages FDA, SWDE, Based SQuAD, and Based NQ. These complementary interventions link the observed depth-wise organization to retrieval performance. §§4.1–4.3 and Appendix E.3 discuss the observed lifecycle, mechanism analysis, and interventions.

Q2. Do soft caps change PAS, ISP, and model quality?

We train a matched pair of 392.9M, 4:1 models from scratch on 10B tokens. SiTU-GLU smoothly caps the gate and up branches before the down projection, while the control uses SwiGLU. Peak denotes the first token's largest pre-FA activation, SSA measures peak alignment, and ISR measures relative plateau retention. Model quality is assessed using ten-task mean accuracy and WikiText PPL.

MeasureSwiGLUSiTU-GLU
Absolute FA peak261.32241.26
PAS alignment99.09%98.96%
ISP retention0.85030.8411
Ten-task mean49.39%44.77%
PPL16.707916.9549

Soft caps reduce absolute peak magnitude, while PAS alignment remains high and substantial ISP retention persists. The capped model also shows lower task accuracy in this matched comparison. These results indicate that attenuating MLP activations alone does not eliminate the hybrid depth pattern. Appendix D.6 provides the training results.

Q3. How does hybridization shape PAS and ISP?

Our controlled experiments vary FA placement, density, and output gating. Moving FA shifts the spike locations; increasing FA density connects isolated spikes into plateaus; and output gating attenuates the peaks. These results show how hybrid architecture shapes the location and persistence of PAS and ISP.

Our write–sink–cancel account provides a unified interpretation: a preceding MLP writes a dominant component, FA reads from the resulting sink, and subsequent opposing updates attenuate or cancel the component. When another FA occurs before cancellation is complete, the component can persist across intervening linear layers, forming an ISP. This connects isolated spikes and sustained plateaus within a common activation lifecycle. Fixed-coordinate tracking further shows persistence within a stable dominant coordinate set, consistent with this interpretation. Appendices E.1 and E.3 elaborate on this account alongside the architectural controls.

AuthorsOpenReview ↗

Response to Reviewer Y9eM (Part 2/2)

We continue with additional controls and the practical implications of our connection method.

Q4. Does deletion isolate magnitude, localization, and normalization?

At the tested FA inputs, even the four largest eligible native non-PAS coordinates cannot match the PAS deletion L2 magnitude within ±5%. The tested neighboring layers likewise admit no strictly magnitude-matched native complete-deletion schedule. To isolate the effect of RMS rescaling, we fully zero the PAS coordinates while fixing the local RMS denominator at its pre-deletion value, thereby preserving the other normalized coordinates at that site. Recall uses the four tasks described in Part 1.

ModelNormalOrdinary zeroFixed-RMS zero
340M28.91%4.49%15.23%
1.3B47.07%28.91%22.27%

Substantial recall degradation remains under fixed RMS, showing that the suppression effect cannot be explained solely by shared denominator rescaling. Ordinary deletion also reduces PIQA/ARC-Easy mean accuracy from 62.85% to 58.33% / 72.80% to 57.92%, and increases PPL from 14.61 to 51.97 / 9.50 to 14.73. Random-four deletion remains close to normal performance. Appendices F.5–F.6 report these controls. We acknowledge that magnitude and localization are not fully disentangled.

Q5. What drives connection gains, and what about deployment?

We separately evaluate interior activation editing, endpoint restoration, and their combination. The table reports mean recall accuracy.

ModelNormalInteriorEndpointsCombined
340M28.91%33.98%28.91%33.98%
1.3B47.07%50.59%47.07%49.80%

Interior editing provides the recall gains, while endpoint restoration alone reproduces normal performance. These results motivate our focus on interior-only editing in §4.3 and Appendices F.1/F.4.

The originally submitted Appendix F.3, Table 20 (now Table 25) evaluates connection on Qwen3.5-27B across reasoning, math, and coding benchmarks (accuracy %):

BenchmarkNormalConnection
GPQA-Diamond77.7878.79
MATH50098.2098.40
LiveCodeBench87.1188.09
AIME202580.0080.00
HumanEval96.9598.17

For B1, with 2,002 input tokens and 1,024 output predictions, we measure the runtime of the original connection method after reference-plan preparation:

ModelPrefill ms, normal→connectPrefill+decode s
340M82.58→84.3639.07→38.98
1.3B90.92→92.9239.49→39.57

Prefill latency increases by approximately 2.2%, while total inference time differs by at most approximately 0.23% in this setting. These measurements exclude reference-plan preparation and provide an initial assessment of the runtime overhead. Further optimization of activation editing and evaluation across broader tasks and settings remain important steps toward practical deployment.

We hope these additions and clarifications address the reviewer's concerns.

Reviewer_SA6oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the massive activations phenomenon in hybrid attention models, which mix linear and normal attention layers. In across five different types of linear attention, two different model sizes, and several hybridization ratios, the authors found a common pattern of massive activation spikes. These massive activations occur before each full attention layer but get canceled out after it. When full attention layers are packed closer together, the massive activations stay up most of the time, instead of spiking and falling. This corresponds to the observation in pure transformers. The same pattern appears in 12 public hybrid models, with one up to 397B. For functional intervention, by zeroing out the top four coordinates of the first token before each full attention layer, accuracy is hurt by 21.9%–63.6% in relative terms. In contrast, holding those coordinates the same between full attention layers gives small relative gains in retrieval accuracy (1.1%–12.6%) without retraining.

Critical strengths

Hybrid linear attention is becoming very standard for today’s models, but the study of the massive activations topic and attention sinks still mostly comes from pure transformers. So the study of it in such hybrid models alone makes the topic important. Beyond the importance itself, the work is strong in several ways.

  • The spike before attention finding rethinks massive activations as something especially for softmax, not just something from generic models. If this holds, this will be a new conceptual contribution.
  • The retrieval connection is practical, since it brings actual performance change, and it can give new direction to hybrid design.
  • The controlled pretraining of gated DeltaNet study in this paper is well designed and it demonstrates the same observation across all models, reducing the suspicions that it is a training outlier.
  • The model selection (5 types + 12 checkpoints) is broad enough.
Critical weaknesses

The paper misses some prior work that can already explain the observations. Besides, the functional claim would need further experimentation support to be solid.

  • For example, Gu et al. (ICLR 2025) showed that removing the softmax can get rid of the sink and massive activation both. Bondarenko et al. (NeurIPS 2023) traced the activation back to softmax as well. Spikes only before softmax layers is what expects, citing and differentiating with them would be helpful.
  • The intervention can be more controlled. The authors zeroing coordination before norm can be a big OOD edit. Intuitively, the damage can be just pushing the internal representation out of the comfortable distribution of the model. It is not equivalent to removing something functionally meaningful. The connection gains are small as well with no control edit as well. And if keeping values high helps retrieval, why does training learn to cancel them?

Gu et al., When Attention Sink Emerges in Language Models (arXiv 2410.10781) Bondarenko et al., Quantizable Transformers (arXiv 2306.12929)

Questions for authors
  1. Does suppression drops survive a mean-value or a random-coordinate control?
  2. Does connection gain hold against a control edit on random coordinate or a non-sink token
  3. If keeping the value helps retrieval, why training learn to cancel them
  4. How do your findings relate to existing work on this topic?
  5. Can you provide alignment and retention metric for the 12 large models?
2 comments
AuthorsOpenReview ↗

Response to Reviewer SA6o (Part 1/2)

We thank the reviewer for recognizing our systematic analysis and for the constructive questions. We provide additional controls for PAS suppression and connection, and quantify PAS and ISP across 12 large language models.

Q1. Do mean replacement and random deletion reproduce the PAS deletion loss?

On the M-A-P GDN 340M/1.3B hybrids (3:1), we zero the first token's four largest-magnitude coordinates at each FA input, replace them with their channel means over the unedited prompt, or zero four random coordinates outside the local top four. Recall is mean accuracy over FDA, SWDE, Based SQuAD, and Based NQ; commonsense is mean accuracy over PIQA and ARC-Easy (equal task weights). PPL is WikiText-103 continuation perplexity (lower is better).

Model / metricNormalTop4 zeroMean fillRandom4 zero
340M Recall mean (%)28.914.4921.8829.30
340M Commonsense mean (%)62.8558.3359.0562.72
340M PPL14.6151.9719.9614.61
1.3B Recall mean (%)47.0728.9121.8847.07
1.3B Commonsense mean (%)72.8057.9265.2272.80
1.3B PPL9.5014.7314.889.50

Mean replacement also causes substantial degradation without zeroing: recall drops by 7.03/25.20 points (paired 95% CIs [3.39, 10.74] / [20.38, 29.73]), commonsense accuracy decreases, and PPL increases. In contrast, random-four deletion remains close to normal performance. These controls provide further evidence for the importance of PAS in §4.3 (Appendices F.4 and F.6).

Q2. Do random-coordinate or non-sink connections bring similar gains?

We apply the same connection rule to the target coordinates, random coordinates, and a non-sink token, using their respective normal trajectories, floors, and endpoints. The random control averages four draws with the same number of coordinates as the target union; the non-sink token lies outside the attention top three and uses its own coordinate union. Metrics are as above.

Model / metricNormalTargetRandomNon-sink
340M Recall mean (%)28.9133.9828.8629.10
340M Commonsense mean (%)62.8563.0062.7262.68
340M PPL14.6114.6014.6414.61
1.3B Recall mean (%)47.0749.8047.1746.88
1.3B Commonsense mean (%)72.8072.8272.8072.73
1.3B PPL9.509.509.509.50

Both controls remain close to normal performance. Targeted connection improves recall over random connection by 5.13 [2.54, 7.83] / 2.64 [0.61, 4.81] points, and over non-sink connection by 4.88 [2.39, 7.47] / 2.93 [0.76, 5.22] points (paired 95% source-cluster intervals). Commonsense accuracy and PPL change little. These results support the importance of target selection in our connection design (F.4, F.6).

Q3. Why does training learn cancellation if persistence helps retrieval?

We thank the reviewer for this thoughtful question. The next-token prediction objective provides sparse and indirect supervision for recall, so not every update learned during pretraining necessarily improves retrieval. A component may be written for an FA readout and subsequently cancelled, even though preserving it for later readouts could benefit retrieval.

Our connection experiment supports this interpretation: extending persistence improves recall beyond the model's native computation. Thus, learned cancellation and improved retrieval through intervention are not necessarily contradictory. We clarify this interpretation in §4.3 and Appendix E.3.

AuthorsOpenReview ↗

Response to Reviewer SA6o (Part 2/2)

We continue with the relationship to prior work and cross-model evidence.

Q4. How do the findings relate to prior work?

Gu et al. (ICLR 2025) study attention-sink emergence and its relationship to key bias and softmax. Bondarenko et al. (NeurIPS 2023) relate outliers to attention heads learning no-op or partial updates. Our contribution focuses on how these activations are organized across depth in hybrid architectures: isolated spikes at sparse FA boundaries, plateaus across linear layers, and their dependence on FA placement, density, and gating. Our write–sink–cancel analysis and targeted interventions further link this architecture-dependent organization to retrieval behavior. We have added these distinctions and citations to Appendix A.

Q5. How consistent are PAS and ISP across the 12 large models?

Due to space constraints, the submission primarily presented representative activation trajectories. We now quantify PAS and ISP for the sink and two non-sink tokens across five natural domains. SSA measures pre-FA peak alignment, peak excess measures local prominence, ISR measures relative retention, and absolute height measures raw plateau magnitude. The table reports five-domain averages for Qwen397B (95% source-cluster CIs in brackets).

TokenSSA (%)Peak excessISRAbsolute height
Sink99.21 [99.02, 99.40]0.970.7423.75 [23.61, 23.89]
Early non-sink67.83 [67.19, 68.34]0.280.885.69 [5.61, 5.74]
Middle non-sink65.99 [65.48, 66.51]0.230.905.69 [5.63, 5.74]

Sink tokens exhibit pronounced PAS and ISP patterns, with higher absolute plateau levels than both controls across all 12 models. Non-sink spikes are generally weaker, while low, relatively flat trajectories can still yield high relative ISR. These observations are consistent with the M-A-P results. §3.4 and Appendix C.3 provide the full comparisons.

We hope these additional analyses and clarifications address the reviewer's concerns.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion period draws to a close, we sincerely thank you for the time and effort devoted to reviewing our work and for your constructive feedback. We hope that the clarifications and additional results in our responses have helped make the findings, scope, and contributions of our study clearer.

We would be grateful for any further feedback on whether our responses address your concerns. If any questions remain or any points would benefit from further clarification, we would be happy to discuss them and provide additional details.

Thank you again for your careful consideration and helpful suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.