acceptodds
Under review as a conference paper at ICLR 2027

Where Massive Activations Come From: Causal Evidence from MLP Geometry in Large Language Models

Abstract

Large language models (LLMs) commonly exhibit massive activations (MAs) across Transformer architectures. MAs are sparse hidden-state coordinates whose magnitudes greatly exceed typical activations and concentrate at specific tokens and dimensions. These extreme values complicate quantization, compression, and efficient inference, yet their generation mechanism remains unclear. Existing studies mainly associate MAs with attention sinks or mitigate them as numerical outliers. However, these approaches do not explain how MAs are generated inside the network. In particular, the causal connection between token-level triggers, layer-wise sources, and weight-space geometry remains unresolved. We address this gap through a systematic study of 26 language models combining component ablations, singular value decomposition, and causal interventions. Our results show that MAs primarily arise from MLP down-projections, where function-token representations align with load-bearing right-singular directions that amplify and write extreme activations, while attention mainly regulates this pathway. These findings establish MAs as structured geometric events and provide a causal account of their emergence across architectures.

Primary area:
interpretability and explainable AI
Keywords:
Massive Activations, Large Language Models, Mechanistic Interpretability, MLP Geometry, Causal Intervention, Singular Value Decomposition.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UUguOpenReview ↗
Rating
4(+1)
Confidence
3(−1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the mechanisms underlying massive activations (MAs) in large language models by analyzing 26 models using component ablations, singular value decomposition (SVD), and causal interventions. The authors identify the MLP down-projection as the primary source of MAs in most models and show that function-token representations preferentially align with the right-singular directions of the down-projection matrix. These aligned directions amplify and accumulate contributions at a small number of output coordinates, while attention primarily regulates this process. The authors further investigate both single-layer and multi-layer generation mechanisms and use targeted interventions to provide causal evidence for the role of the identified geometric structure. The results offer a unified explanation of MAs across a range of model architectures.

Critical strengths
  1. The paper provides a systematic investigation of MA generation by connecting token-level triggers, layer-wise sources, and weight-space geometry. The combination of component ablations, frequency-controlled token analysis, SVD-based decomposition, and causal interventions offers a coherent approach to understanding the underlying mechanism.

  2. The experiments cover 26 language models across multiple architectural families. The distinction between single-layer and multi-layer source regimes is particularly useful, and the causal interventions provide evidence beyond simple correlations between activation magnitudes and singular directions.

Critical weaknesses
  1. The definition of function tokens combines several heterogeneous categories, including conventional function words, punctuation, structural markers, digits, and short BPE fragments. These categories may exhibit different activation patterns for different reasons. Consequently, the observed function-token enrichment may partly depend on the operational definition, making it difficult to determine how much of the effect is specifically attributable to linguistic function words rather than structural tokens or tokenization artifacts.

  2. The causal interventions establish that perturbing the identified singular-vector geometry can substantially suppress MAs, but their specificity remains insufficiently characterized. Randomizing the right-singular-vector matrix changes the weight matrix globally, and removing selected directions may also affect other model computations. Without evaluating the impact on ordinary model behavior and comparing against carefully matched control interventions, it remains unclear how selectively these interventions target the MA-generating mechanism.

  3. The main experiments use relatively short English text windows, and the independent-document robustness analysis covers only a subset of the evaluated models. Although the results are encouraging, their generalization to other corpora, languages, longer contexts, and different token distributions remains insufficiently established.

Questions for authors
  1. The authors should incorporate and discuss the related work https://arxiv.org/abs/2510.10238. A more explicit comparison would help clarify the novelty of the proposed SVD-based geometric explanation relative to existing findings on critical neurons and MLP down-projection components. In particular, the authors should explain whether the identified right-singular directions provide a more specific mechanistic account of MA generation beyond identifying important neurons or layers.

  2. The authors should conduct additional ablations to disentangle the contributions of different function-token categories. For example, conventional function words, punctuation and structural markers, digits, and short BPE fragments could be analyzed separately. Repeating the frequency-controlled analysis under these alternative definitions would help establish whether the reported token preference is robust and linguistically meaningful across model tokenizers.

  3. The authors should strengthen the causal validation with matched control interventions, such as removing random directions with comparable dimensionality or norm, and compare targeted interventions against leading-direction ablations. They should also evaluate language-model perplexity and downstream task performance before and after intervention to determine whether MA suppression can be achieved without broadly disrupting model functionality.

2 comments
AuthorsOpenReview ↗

Response on Function-Token Mechanisms, Performance Effects, and Generalization

Thank you for the review.

(RevC-1) Contribution beyond critical-neuron studies

Our work adds a quantitative connection between function-token (FT) representations, singular directions, actual MA-coordinate writes, and layer-dependence patterns. Qin et al.'s critical-neuron study identifies influential MLP components through intervention.

At each token, we measure norm, normalized alignment, projection, and output. The contribution of direction ii to coordinate jj is ci(t,j)=σi(vi⊤h2(t))ui[j]c_i(t,j)=\sigma_i(v_i^\top h_2(t))u_i[j]: viv_i determines input projection, σi\sigma_i amplification, and ui[j]u_i[j] output localization. We test low-order reconstruction against actual writes, including signed cancellation, bias, and higher-order contributions.

Across 26 models, per-layer/cumulative MLP ablations and recovery organize Concentrated (9), Few-Source (7), Dispersed (8), and Anomaly (2) patterns. Representative-model macro measurements additionally connect calibrated output directions to fixed-coordinate increments and actual layer writes. A concentrated direct writer can coexist with dependence on other layers. Directional interventions and matched controls connect this geometry to selective MA responses, yielding a quantitative taxonomy of MA activation patterns.

(RevC-2) Why combine different FT categories?

FT is the operational token category used to quantify the relation between token representations and MA writing. It combines conventional function words, punctuation/structural markers, digits, and short BPE fragments, including BOS.

The paired geometry experiments use Qwen2.5-7B-Instruct, Mistral-7B-v0.3, and Llama-3.1-8B-Instruct on 256 C4 documents of 256 tokens each. At the measured amplification blocks, FT accounts for 147/258, 503/503, and 256/256 strong events, respectively (same-token residual peak/median ratio ≥300\ge 300). We measure norm, normalized direction, and actual coordinate writing separately; the observed model differences are retained. Our present focus is their quantitative correspondence and single-/multi-layer dependence. Systematic causal separation of FT subclasses and tokenizer effects remains future work.

(RevC-3) Can MA suppression and performance changes be distinguished?

MA-peak suppression and prediction-performance changes are separable under the tested interventions.

We tested 43 conditions in Q4 = Qwen3-4B, Q8 = Qwen3-8B, and L8 = Llama-3.1-8B-Instruct, using 128 C4 documents of 256 tokens/model. Directions and observation blocks are fixed before evaluation; we track the baseline token/coordinate.

First, removing 50% of the current target output-direction component gives:

ModelOriginal MA peak changePPL beforePPL after
Q4-49.952%22.24822.254
Q8-74.553%17.36017.207
L8-50.000%11.91011.848

Second, we compare target and orthogonal random directions at a separate 25% baseline-referenced activation budget. All five seeds match each intervened layer's aggregate actual output-change norm within 5%:

ModelTarget peak changeRandom peak rangeTarget PPLRandom PPL range
Q4-24.954%0.000%22.26031.090-37.834
Q8-47.636%-0.499% to +0.018%17.22820.458-23.023
L8-24.845%-1.242% to +0.621%11.83618.941-20.410

Random columns cover all five seeds. Random updates increase PPL while preserving most of the original peak; target updates suppress it.

We also tested full target-direction removal in the weights on 256 PIQA questions, scoring continuations by mean token log-likelihood. The 95% intervals use 2,000 paired question-bootstrap draws:

ModelAccuracy before/afterChange in percentage points [95% CI]
Q471.48%/57.42%-14.06 [-20.31, -7.81]
Q873.44%/71.48%-1.95 [-7.04, +3.52]
L881.25%/53.52%-27.73 [-35.16, -20.70]

Complete removal has performance costs; the small-PPL results concern partial intervention.

(RevC-4) Transfer to other corpora and longer contexts

The C4-calibrated intervention transfers to WikiText and the tested 1,024-token windows, suppressing the original MA peak by over 99.8% in all three models.

We applied C4-calibrated directions, layer sets, and intervention weights directly to WikiText. We evaluated 128 documents at 256 tokens and paired the first 64 at 256 and 1,024 tokens. The table reports the complete short-window sample and the long-window subset:

Model256 tokens/128 documents: peak change1,024 tokens/64 documents: peak change
Q4-99.834%-99.833%
Q8-99.982%-99.982%
L8-99.922%-99.922%

Broader corpora, languages, context lengths, and token distributions are future work.

I appreciate the authors' thorough response. Their clarifications have completely resolved my initial concerns, and I am happy to increase my score accordingly.

Reviewer_hSCuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies massive activations across 26 open models, including dense, hybrid-attention, and MoE checkpoints up to 35B total parameters. Component ablations distinguish MLP generation from attention regulation, while function-token analysis and SVD examine token triggers and projection geometry. Randomization and directional removal provide intervention evidence, with a macro-direction extension for multi-layer sources. The proposed causal account connects token representations to extreme residual coordinates; appendices report exceptions and sampling sensitivity.

Critical strengths
  1. Broad empirical coverage. MLP-output ablation suppresses massive activations in 23/26 models, while attention removal has heterogeneous effects (§4.1–4.2). The function-token analysis attempts to control frequency rather than relying only on correlation. The appendix also discloses exceptions and trigger-layer sensitivity in the independent-document rerun.

  2. A useful geometric intervention. Haar randomization of V preserves the singular values and output-side directions of W_down while changing its input orientation (Eq. (8)). This adds information beyond zeroing the MLP branch, since it preserves matrix scale. The conditional prediction in Eq. (9) makes the account quantitatively testable.

Critical weaknesses
  1. The macro intervention and causal reporting are not sufficiently clear. Eq. (2) defines W_down as D×d_ff and h2 as d_ff-dimensional. Eq. (10) then right-multiplies W_down by a projector built from the SVD of a residual-stream delta. As written, a residual-space feature direction is D-dimensional, so an additional mapping into each MLP's input space is required. Eq. (15) raises the same issue. This may be an omitted implementation detail, but it affects the central multi-layer result. Furthermore, §4.6 says macro projection preserves each layer's singular values and left singular vectors. Direction removal generally changes these quantities; spectrum preservation applies to the random-V replacement, not to the stated projection operator.

    For causal interventions, Appendix B.3 combines multi-K selection, a near-threshold allowance, and a signed-coordinate clause that can count a sign reversal without reducing absolute magnitude. Appendix C.2 already reports strict dashboard rates of 8/9, 7/7, 4/8, and 0/2. These need reconciliation with §4.6's 12/12 clean dense collapse claim, particularly Qwen1.5-14B's macro change of −12.6%. Fig. 5 also labels RQ6 as “Recovered (keep best 1 MLP),” which is a different intervention from macro-direction removal.

  2. The token-specific mechanism needs stronger quantitative support. Randomizing V changes the learned map for all tokens. The paper does not report non-MA activation changes or held-out perplexity under this intervention. Large raw projections can also reflect large intermediate norms or a few outlier coordinates, rather than unusually strong angular alignment. Selected-direction removal is more specific, but random-subspace comparisons and a decomposition of norm, alignment, and output amplification are needed to support the stronger causal interpretation.

    Eq. (6) is an exact SVD identity; the empirical limitation is its low-rank explanation. At k=20, truncation error still exceeds 30% in 6/22 analyzed models (§4.4). The claimed coherent contribution signs in Eq. (14) need direct measurements; the separate 16/22 source-layer recovery statistic does not measure sign consistency.

    Theorem D.5 factors the activation magnitude into a projection term while omitting b_down, although the proof says bias is preserved. It also predicts a fixed-coordinate, rank-one quantity; Appendix B.1 defines Top1 as a maximum across tokens and coordinates. The comparison should use the same measured object and quantify bias, remaining singular terms, and normalized alignment.

Questions for authors
  1. What matrix shapes and mappings are used in the implemented macro SVD and projection? Please reconcile Eqs. (10)/(15) and the spectrum-preservation statement in §4.6.
  2. How do non-MA activations and perplexity change? Compare leading-K removal with random K-dimensional input-subspace removal, reporting removed weight energy and overall distortion, and include comparable non-source-layer perturbations.
  3. Which D1–D4 clause does each model pass? Please distinguish signed-coordinate changes from absolute-Top1 suppression and identify the measurements plotted in Fig. 5.
  4. Can you report termwise sign agreement and cancellation for the models with poor low-rank fits, separately from source-layer recovery? Can the Eq. (9) prediction be compared with the identical fixed token/coordinate statistic under explicit bias assumptions?
  5. Does enrichment persist after excluding BOS and special/chat markers? A token-class decomposition of intermediate norm, normalized singular-direction alignment, and coordinate contributions would clarify what the function-token label explains.
2 comments
AuthorsOpenReview ↗

Response on Macro Interventions, Success Criteria, and Geometric Evidence

Thank you for the constructive review.

(RevB-1) Macro space, projection, and spectral preservation

Macro qq lies in residual-output space, so the right-multiplication in Eqs. 10 and 15 was incorrect. It is the leading right singular direction of ΔH=Hout−Hin∈RN×D\Delta H=H_{\mathrm{out}}-H_{\mathrm{in}}\in\mathbb{R}^{N\times D}, with q∈RDq\in\mathbb{R}^D. For Wℓ∈RD×dffW_\ell\in\mathbb{R}^{D\times d_{\mathrm{ff}}}, the correction is Wℓ′=(ID−qq⊤)WℓW'_\ell=(I_D-qq^\top)W_\ell. Bias is unchanged. This projection generally changes singular values and left singular vectors; preserving UU and Σ\Sigma applies to full random-VV replacement.

Hooked and explicit projections agree; restoring weights restores outputs. Directions, coordinates, and rank 1 were fixed on 32 C4 calibration documents. Evaluation uses 256 documents of 256 tokens; errors concern strong events (peak/median ≥300\ge 300):

ModelIntervened blocksFixed coordinateRank-1 increment error
Q4{6,15}40.021526%
Q8{0,2,3,5,6,7}22760.016147%
L8{0,1}7880.000341%

Q4/Q8/L8 are Qwen3-4B/Qwen3-8B/Llama-3.1-8B-Instruct; blocks are zero-indexed. Errors concern fixed-coordinate increments; MLP/attention writes and BF16 rounding are recorded over intervals 6-15/0-7/0-1. In Q4, deleting blocks 6 and 15 separately reduces the global extreme by 20.38% and 3.85%; deleting both reduces it by 89.14%. Thus, a dominant direct writer can still depend on other layers.

(RevB-2) Can suppression be separated from general disruption?

Targeted suppression of the original MA peak is separable from general prediction-performance deterioration.

We tested 43 conditions on 128 C4 documents/model (256 tokens), tracking the baseline peak token/coordinate. Removing 50% of the current target component gives:

ModelMA peak changePPL beforePPL after
Q4-49.952%22.24822.254
Q8-74.553%17.36017.207
L8-50.000%11.91011.848

At a separate 25% baseline-referenced budget, five qq-orthogonal random seeds match actual per-layer aggregate activation changes within 5%. Target peak changes are -24.954%/-47.636%/-24.845%; random ranges are 0%/[-0.499%,+0.018%]/[-1.242%,+0.621%]. Random PPL rises to 31.090-37.834/20.458-23.023/18.941-20.410, versus target PPL 22.260/17.228/11.836.

With full removal on 256 documents, off-mask relative L2 changes are 0.989, 0.964, and 0.998; PPL becomes 75.930, 28.552, and 468.640.

(RevB-3) Success criteria, model counts, and Fig. 5

At an 80% mean-magnitude reduction threshold, the historical screen gives 19/26 models (73.08%): 18 supported by raw files plus BLOOM's historical 82% aggregate.

We separate full random-VV replacement from D1 top-KK deletion; report KK and the intervention/observation layers; distinguish signed values from magnitude in D2; use the actual macro layer set in D3; and keep D4 borderline cases outside the strict 80% count. Missing coordinates are not zero. Fig. 5 measures tested single-MLP recovery, not macro deletion.

The other 7 models (26.92%) have these standard single-layer/macro reductions:

ModelSingle-layer / macro reduction
Qwen2.5-0.5B55.04% / unverified
Qwen3.5-9B70.18% / 0.25%
Qwen1.5-14B49.29% / 12.63%
Qwen3.5-27B77.93% / 0.38%
Qwen3-30B-A3B0.85% / 0.37%
OPT-6.7B17.55% / unverified
Qwen3.5-35B-A3Bincreases 0.05% / 0.52%

Separate protocols give Qwen1.5-14B fixed-dimension reductions of 96.58-98.27%; Qwen2.5-0.5B's two dimensions fall by 42.18-49.75% and 75.65-78.08%; per-expert Qwen3-30B-A3B gives 91.04%/96.16%. Historical counts are unchanged. We have corrected the statistics and the 12/12 macro claim.

(RevB-4) What supports low-rank explanation?

Sign consistency and low-rank accuracy are distinct: the Q8/L8 examples have little cancellation but substantial 20-term error. We recomputed 20 signed terms per record across 22 models (1,000 records/model). At the largest exported row, Q8/L8 cancellation is 0.00368%/about 1.52%, but K=20K=20 amplitude error is 67.60%/88.07%. Macro mean largest-layer write shares are 97.47-99.98%, with inter-layer cancellation of 0-0.94%, consistent with concentrated writing and multi-layer dependence. Eq. 9/D.5 now specifies the same token/layer/coordinate and accounts for bias, higher-order terms, normalization, and alignment.

(RevB-5) Norm or directional alignment?

Qwen's large projection is not a norm-only effect; its normalized representation remains aligned with v1v_1.

On 256 C4 documents/model, we measure norm, normalized v1v_1 alignment, and signed reconstruction for broad-FT strong events (residual peak/median ≥300\ge 300):

Model/blockMean normMean absolute cosineK=20K=20 error
Qwen2.5-7B-Instruct/3931.690.642740.756%
Mistral-7B-v0.3/11014.590.1575840.827%
L8/1480.900.0326788.009%

FT includes BOS; detailed mechanisms within its subclasses remain future work.

AuthorsOpenReview ↗

Request for feedback

Dear Reviewer,

As the discussion period is coming to an end, we would appreciate your feedback on our rebuttal and additional experiments.

We are happy to address any remaining concerns, but we may not have the opportunity to respond once the discussion period closes.

Thank you for your time.

Reviewer_d89DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates massive activations (MAs) across 26 open LLMs through attention and MLP ablations, token analysis, SVD, and causal interventions. It finds that MAs are mainly generated by low-rank structures in early-layer MLP down-projections aligned with function-token representations, while attention primarily modulates this pathway. Randomizing or removing the relevant singular directions suppresses MAs by 93–99% in tested models, supporting the proposed mechanism across both single- and multi-source regimes.

Critical strengths

Coverage is unusually broad (26 models, several families and scales), and the regime taxonomy is reported honestly, including the ANOMALY cases and the 3 models where MLP ablation does not suppress MAs.

The work moves beyond correlation toward weight-level analysis (SVD of W_down, structured replacement of singular vectors), which is the right direction for a mechanistic account.

Critical weaknesses

The causal tests lack specificity, and the criteria are loose. The paper needs (a) the same intervention on non-MA coordinates or layers, (b) the perplexity or downstream cost of each intervention, and (c) rotating only the non-top singular directions while keeping v₁ fixed. Without these, "MAs disappear" cannot be separated from "the layer is broken."

"The Super Weight in LLMs" (Yu et al., 2024) already traced super activations to individual down_proj weights in early layers, and Sun et al. (2024) already localized MAs to early-layer MLPs. Neither is adequately discussed or compared against.

The frequency-controlled logistic regression reports neither β_F nor its confidence interval.

Questions for authors

In RQ5, what happens to MA magnitude and perplexity under random replacement of (a) a non-MA layer and (b) only the non-top singular directions?

Which value is the correct function-token baseline, ~30% or ~53%? And why do the trigger layers move in 5/7 models when the documents change?

2 comments
AuthorsOpenReview ↗

Response on Intervention Specificity, Novelty, and Function-Token Analysis

Thank you for the constructive review.

(RevA-1) Targeted MA suppression or general damage?

The intervention is targeted: suppression of the original MA peak is not a nonspecific consequence of general model damage.

We tested three models: Q4 = Qwen3-4B, Q8 = Qwen3-8B, and L8 = Llama-3.1-8B-Instruct. Original-peak measurements track the same baseline token/coordinate at observation blocks 15/7/1. A 43-condition study uses 128 C4 documents per model, each with 256 tokens. Removing 50% of the current target output-direction component gives:

ModelOriginal MA peak changePPL beforePPL after
Q4-49.952%22.24822.254
Q8-74.553%17.36017.207
L8-50.000%11.91011.848

For direction specificity, we preserved the weight matrix's mapping along output direction qq and perturbed its orthogonal complement with five random seeds. On 256 C4 documents/model, target removal reduces the peak by 99.826%/99.961%/99.922%; random peak changes are 0%/[-0.114%,+0.219%]/0%, respectively.

We also compared target and qq-orthogonal random updates at a 25% baseline-referenced activation budget on 128 documents. All five seeds match each intervened layer's aggregate actual output-change norm within 5%:

ModelTarget peak changeRandom peak rangeTarget PPLRandom PPL range
Q4-24.954%0.000%22.26031.090-37.834
Q8-47.636%-0.499% to +0.018%17.22820.458-23.023
L8-24.845%-1.242% to +0.621%11.83618.941-20.410

Ranges cover all five seeds. Random updates worsen PPL while largely preserving the peak, supporting the tested direction's selective effect. Complete removal has larger performance costs: PIQA accuracy changes from 71.48% to 57.42%, 73.44% to 71.48%, and 81.25% to 53.52%. The small-PPL result concerns partial interventions.

(RevA-2) What is added beyond MA and Super Weight studies?

Our contribution is the quantitative connection between function-token (FT) representations, singular directions, actual coordinate writes, and cross-model layer dependence. Sun et al. established key MA properties and attention effects; The Super Weight in LLMs traced large activations to down-projection weights.

For the same token, we measure norm, normalized alignment, raw projection, and the signed contribution ci(t,j)=σi(vi⊤h2(t))ui[j]c_i(t,j)=\sigma_i(v_i^\top h_2(t))u_i[j]. This separates input projection, amplification, and output-coordinate localization. We compare low-order reconstruction with actual MLP writes and measure cancellation and higher-order contributions.

We also organize 26 models into Concentrated (9), Few-Source (7), Dispersed (8), and Anomaly (2), using per-layer/cumulative ablations and recovery. Macro measurements connect an aggregate output direction to fixed-coordinate increments and actual layer writes. Together, these results provide a systematic description of MA activation patterns: concentrated direct writing can coexist with dependence on other layers.

(RevA-3) Frequency-controlled coefficients and confidence intervals

FT remains positively associated with the defined high-activation events after frequency adjustment.

The original analysis found FT enrichment at MA-triggering positions and a positive FT association after frequency control. Following your suggestion, we report coefficients and 95% intervals for three representative models, each with 256 C4 documents and 65,536 tokens. For exploratory top-1%-magnitude events at a fixed observation layer, we fit logistic regression with FT membership and same-corpus log token frequency, using document-cluster-robust intervals:

ModelFT coefficient95% CI
Qwen2.5-7B-Instruct1.135[0.926, 1.345]
Mistral-7B-v0.32.088[1.653, 2.523]
Llama-3.1-8B-Instruct1.043[0.691, 1.394]

All three coefficients are positive, with 95% intervals excluding zero. This also holds using calibration-set frequency estimates.

(RevA-4) Why 30% versus 53% FT background?

All models use the same baseline rule: FT tokens / all candidate tokens in their text samples, with tokenizer and FT definition matched to the trigger analysis. The 30% and 53% values refer to 64 C4 documents of 256 tokens and 1,000-token WikiText-2 windows (52-59%), respectively. We have labeled these settings and cited the appendix's Activation and Function-Token Metrics / Base rate paragraph.

(RevA-5) Why do reported layers change with documents?

The reported changes concern document-specific peak layers, which need not coincide with the MLP writing/amplification layers studied here. We distinguish initial signal, amplitude surge, MLP writing, ablation sensitivity, and peak observation. The cited appendix selects the maximum-activation layer; our mechanism analysis concerns writing/amplification and single-/multi-layer dependence. We have clarified these terms; systematic peak-layer migration is reserved for future work.

AuthorsOpenReview ↗

Request for feedback

Dear Reviewer,

As the discussion period is coming to an end, we would appreciate your feedback on our rebuttal and additional experiments.

We are happy to address any remaining concerns, but we may not have the opportunity to respond once the discussion period closes.

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.