acceptodds
Under review as a conference paper at ICLR 2027

How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing

Abstract

Hyper-Connections and their manifold-constrained variant mHC widen a residual pathway from one stream to , yet how trained models use this capacity remains unclear: how broadly blocks read and write, how strongly the residual pathway mixes streams, and whether the streams carry distinct representations. We examine these properties in the four-stream residual pathway of DeepSeek-V4-Flash using effective stream counts, cross-stream residual weights, and inter-stream cosine similarity. Read/write routing is concentrated but varies across depth: a typical attention or FFN site effectively uses about two streams, while the dominant stream changes across layers and the representations remain directionally distinct. Residual mixing is modest and occurs primarily in early layers; in layers 22–42, the pathway mostly carries each stream forward separately. Targeted interventions establish the functional significance of these patterns. Replacing the late mixers by identity increases C4 perplexity by only and preserves the six-task average score, whereas replacing the early mixers increases perplexity by . Fixing each early mixer to its C4 diagnostic mean increases perplexity by only and reduces the average score by percentage points, showing that its site-specific structure matters more than its token-wise variation on the evaluated metrics. Likewise, retaining the three largest routing weights per token at every site increases perplexity by at most and changes the average score by at most points. Thus, the studied model realizes only part of the flexibility afforded by four-stream mHC: individual blocks rarely require all four streams, and late residual mixing provides little measured benefit.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
large language model, mHC

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_mz1VOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an empirical analysis of how the four streams in mHC function in DeepSeek-V4-Flash. The authors find that a typical attention or FFN module effectively utilizes approximately two streams, with the dominant stream varying across layers, while the stream representations remain directionally distinct. Another key observation is that residual mixing is relatively limited and primarily concentrated in the early layers. In layers 22–42, the residual pathways largely preserve each stream independently.

Critical strengths
  1. The paper provides useful evidence from a large pretrained model. The changing dominant streams across depth show that the single-stream collapse reported in smaller models does not describe every trained HC model.

  2. The fixed-mean mixer experiment is particularly interesting. It separates the importance of the mixing pattern at each site from the benefit of adapting that pattern to individual tokens.

Critical weaknesses
  1. The paper shows that some routing and mixing operations can be simplified in an existing model. It does not report whether these changes save computation or speed up inference, so their practical benefit is still unclear.

  2. All results come from one checkpoint. The differences from earlier small-model studies are interesting, but the paper cannot tell whether they come from model scale, architecture, or training setup. This limits how broadly the findings can be interpreted.

Questions for authors
  1. I found the fixed-mean result interesting: replacing early-layer mixers with their means raises perplexity by only 0.2%, compared with 41% for identity matrices. But this is tested only after training. Could fixed mixers work just as well if used from the start?
3 comments

We thank the reviewer for the constructive feedback and for recognizing the value of our fixed-mean mixer analysis and large-model characterization.

1. Practical Inference Efficiency

We implement the structural simplifications in the official MP8 inference pipeline and benchmark them on 8×H20 GPUs using the same checkpoint and precision.

Implementation. We evaluate four configurations:

SettingLayersImplementation
NativeL0–42Original mHC
Early MeanL0–21Static C4 mean mixers; skip residual-logit computation and Sinkhorn, while retaining dense 4×4 residual mixing
Late IdentityL22–42Replace dense residual mixing with identity addition
CombinedL0–42Apply both simplifications

End-to-end latency. We measure prefill across batch sizes 1/8 and prompt lengths 2K/8K, and decode at batch sizes 1/8 with a fixed 32-step generation. Model loading and tokenization are excluded. We report median latency [min, max] over repeated runs, with percentage changes relative to Native (negative means faster).

Prefill latency (seconds)

WorkloadNativeEarly MeanLate IdentityCombined
B1/2K2.724[2.687,2.809]2.710(-0.52%)2.659(-2.41%)2.718(-0.24%)
B1/8K6.152[6.012,6.212]6.137(-0.24%)5.834(-5.17%)6.028(-2.01%)
B8/2K10.891[10.187,11.185]11.028(+1.25%)10.240(-5.98%)10.559(-3.05%)
B8/8K35.023[34.905,35.329]35.295(+0.78%)32.779(-6.40%)33.024(-5.71%)

Decode latency (seconds, fixed 32 steps)

WorkloadNativeEarly MeanLate IdentityCombined
B17.082[7.080,7.090]7.062(-0.29%)7.001(-1.14%)7.010(-1.02%)
B819.814[19.810,19.866]19.856(+0.21%)19.770(-0.23%)19.884(+0.35%)

Peak allocated memory (GiB)

WorkloadNativeEarly MeanLate IdentityCombined
B1/2K23.81823.81823.78623.786
B1/8K26.58926.58926.46426.464
B8/2K30.28230.28230.03230.031
B8/8K52.44852.44851.44851.444

Summary. Late Identity reduces prefill latency by 2.41–6.40% and peak allocated memory by up to 1 GiB at B8/8K. It also reduces decode latency by 0.23–1.14% across the tested settings. Early Mean maintains near-native end-to-end latency, while combining both simplifications achieves up to 5.71% lower prefill latency.

These results demonstrate practical, workload-dependent inference benefits from simplifying mHC residual mixing. We will include these results in the revised manuscript.

2. Generality Beyond a Single Checkpoint

To examine whether our findings are specific to DeepSeek-V4-Flash-0731, we additionally analyze the DeepSeek-V4-Flash checkpoint. Both checkpoints exhibit highly consistent routing and mixing patterns:

MetricFlashFlash-0731
Mean effective streams (Read)1.89971.8943
Mean effective streams (Write)1.70771.6945
Mean mixer diagonal (L0–21)0.90810.9080
Mean mixer diagonal (L22–42)0.98370.9821

In both checkpoints, each of the four streams is the most frequent routing winner at some sites, with the dominant stream changing across depth. Read/write routing remains concentrated, while mixer diagonals are consistently higher in later layers, indicating weaker residual mixing. Read/write routing remains concentrated, while residual mixing becomes weaker in later layers.

Importantly, DeepSeek-V4-Flash provides a real-world example of mHC deployed in a large-scale, fully trained model, complementing previous studies on smaller models. Our consistent findings across two checkpoints further support the robustness of these observations.

3. Fixed Mixers from the Start of Training

We thank the reviewer for this insightful suggestion. Our fixed-mean experiment demonstrates that token-wise mixer variation has limited impact on an already-trained model, but does not establish whether it is necessary during training. We hypothesize that static, site-specific mixers may also learn effective mixing structures when jointly optimized with the model from initialization. Testing this hypothesis requires controlled from-scratch comparisons between dynamic and static learnable mixers, which we leave for future work. We will clarify this distinction in the revised manuscript.

Reviewer_mz1VOpenReview ↗

Reply by Reviewer mz1V

Thank you for your response.

However, my main concern remains unresolved. It is still unclear how the findings could be used in practice or what concrete benefits they would provide.

I am therefore inclined to keep my current score, leaning toward rejection.

Thank you for the clarification. We further emphasize the practical and architectural implications of our findings.

Practical benefits. Our results provide actionable guidance for simplifying existing mHC checkpoints without retraining. Replacing late mixers (L22–42) with identity reduces prefill latency by 2.41–6.40% on 8×H20 GPUs, with up to 1 GiB lower peak allocated memory. At B8/8K, latency drops from 35.023 to 32.779 seconds. This comes with a measurable 1.23% C4 PPL increase, while no degradation is detected on C-Eval or GSM8K under our evaluated metrics.

Implications for mHC design. More broadly, our findings suggest a depth-aware design principle: preserve site-specific residual mixing in early layers, explore simpler near-identity mixing in later layers, and retain depth-dependent, read/write-specific routing rather than enforcing a globally dominant stream. This suggests that multi-stream computation should be allocated according to its functional importance across depth, rather than uniformly across all layers.

We will incorporate these practical findings and architectural implications into the revised manuscript.

Reviewer_DSuhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how DeepSeek-V4-Flash, a 284B-parameter 43-layer MoE model with four-stream mHC residual pathways, actually uses its multi-stream computation at inference time. The authors instrument 86 attention and MoE-FFN residual sites and perform a “true-forward” analysis of realized read/write routing, residual mixing, and inter-stream representation geometry. They define winner consistency and load-effective stream count for read/write maps, identity deviation for residual mixers, and pairwise cosine similarity for stream representations. They then run inference-time interventions: top-k routing sparsification and residual-mixer replacement by identity or by a C4 diagnostic mean. The main findings are that routing is locally concentrated—typically about two effective streams per site—but the dominant stream changes across depth, sublayer type, and read/write map; stream representations remain directionally distinct; residual mixing is concentrated in layers 0–21 and becomes near identity in layers 22–42. Interventions show that keeping the top-3 read/write weights is nearly lossless, whereas top-2 causes clear degradation; replacing late residual mixers by identity is nearly lossless, while replacing early mixers by identity is very harmful, although replacing early mixers by their site-specific C4 means is nearly lossless. The authors frame this as “structured under-utilization” rather than complete stream collapse.

Critical strengths

(1) The main issue—whether architectural multi-stream capacity is actually used after training—is well motivated. mHC promises additional capacity and cross-stream exchange, but realized computation may be much narrower.

(2) Previous observations of concentrated routing or near-identity residual mixing do not necessarily establish functional redundancy. By explicitly modifying routing maps and residual mixers and measuring the resulting changes in model performance, the authors provide stronger evidence than could be obtained from visualizations alone. This experimental strategy is appropriate for the paper's central objective of connecting learned architectural structure to functional behavior.

(3) The authors explicitly distinguish representational separation from functional importance and acknowledge that training-time trajectories would be necessary to establish stronger explanations of how the observed routing and mixing patterns emerge.

Critical weaknesses

(1) Inference-time ablations show whether removing a component hurts the evaluated metrics, not why training produced that organization. Appendix A gives a local gradient heuristic, but the authors note that it does not reconstruct training dynamics. The main text still uses terms such as “functional localization,” “underused flexibility,” and “structured under-utilization,” which imply mechanistic explanation. These should be strictly framed as inference-time redundancy on the evaluated metrics, with training mechanisms labeled as speculation.

(2) My another primary technical concern is the definition and interpretation of the load-effective stream count. The authors first average normalized routing weights across tokens and then calculate the inverse squared concentration. However, this is not equivalent to averaging the effective number of streams used by individual tokens. The distinction is important because the proposed metric conflates routing breadth within individual tokens with diversity in routing assignments across tokens. For example, consider a setting in which each token routes exclusively to one stream, but the selected stream varies uniformly across four streams. The proposed aggregate metric would report four effective streams even though every individual token uses exactly one. Thus, the reported effective stream counts cannot be directly interpreted as the typical number of streams used by an individual token.

(3) The late-mixer identity intervention is an important result, but its interpretation should be more carefully qualified. The perplexity evaluation uses relatively short C4 sequences, while the downstream evaluations focus on conventional language understanding and reasoning benchmarks. Given the long-context capabilities of the analyzed model, it remains unclear whether late residual mixing is similarly dispensable for long-context or long-horizon tasks. Furthermore, the absence of confidence intervals makes it difficult to claim whether the relatively small changes represent statistical equivalence or simply limited sensitivity of the selected evaluations.

Questions for authors

(1) Could the authors report the average of token-level effective stream counts and compare it with the currently reported effective count of token-averaged routing distributions? Does the conclusion that typical sites utilize approximately two streams remain valid under the token-level statistic? It directly affects the interpretation of the main routing results.

(2) The authors should report paired uncertainty estimates for the C4 evaluation and task-level confidence intervals where feasible. In particular, what evidence supports treating the late-mixer intervention as approximately performance-preserving rather than simply producing a small observed change on the selected benchmarks?

(3) What is the precise novelty relative to prior work? Could the authors more explicitly explain how the observed depth-varying routing regime differs from previously reported globally dominant-stream behavior? In particular, which findings are genuinely new phenomena, and which are large-scale confirmations of existing observations?

(4) Restrict the conclusion to “for this checkpoint and evaluated metrics,” and explicitly label Appendix A as hypothesis-generating.

2 comments

We thank the reviewer for the constructive feedback and suggestions.

1. Scope of Inference-time Interventions

Our interventions test the functional importance of routing and mixing in the trained model, not their training origins. Top-3 pruning and late-mixer identity replacement have limited impact; top-2 pruning and early-mixer identity replacement substantially degrade performance. Site-specific early means preserve most of the early mixers’ measured benefit.

“Functional localization” refers to this intervention-based evidence; “structured under-utilization” describes non-uniform use of four-stream capacity. The appendix’s local gradient analysis is hypothesis-generating.

2. Token-level and Load-level Effective Stream Counts

Our metric in Eqs. (6)–(7) averages routing distributions over each sequence’s TmT_m tokens before computing the effective count, capturing sequence-level routing load. We retain it and add token-level and all-token pooled measures.

For normalized routing distribution pm,tp_{m,t}, we report: NtokenN_{\mathrm{token}} (per-token), NseqN_{\mathrm{seq}} (each sequence-averaged), and NpooledN_{\mathrm{pooled}} (pooling all tokens and sequences).

CIs use 10,000 circular block-8 bootstrap resamples (seed 42).

Map/SublayerNtokenN_{\mathrm{token}} (95% CI)Original NseqN_{\mathrm{seq}}NpooledN_{\mathrm{pooled}} (95% CI)
Read/Attention2.0215 [2.0196,2.0234]2.13972.1451 [2.1407,2.1502]
Read/FFN1.7670 [1.7642,1.7698]1.85541.8599 [1.8557,1.8647]
Write/Attention1.5316 [1.5299,1.5333]1.57051.5735 [1.5717,1.5756]
Write/FFN1.8574 [1.8527,1.8623]1.98051.9884 [1.9776,2.0013]

Token-level effective stream counts average 1.53–2.02 across groups, indicating concentrated routing. Site-level medians further reveal differences across sublayers, ranging from 1.2930 for write-attention to 1.8493 for read-attention. These results further support our original finding of concentrated yet heterogeneous routing across sublayers and sites.

Cross-layer winner dynamics. We pool token-level argmax assignments at each site and track dominant-stream changes across adjacent layers.

Map/SublayerWinner consistencyAdjacent-layer switches
Read/Attention0.848219/42
Read/FFN0.884315/42
Write/Attention0.942717/42
Write/FFN0.861515/42

All four streams emerge as dominant winners at different sites, with Stream 1 preferred at 23 of 43 write-attention sites. This further supports our original finding that stream dominance varies across sites and depth rather than being globally fixed.

3. Paired Uncertainty and Long-context Evaluation

C4. We evaluate 256 held-out documents (524,032 scoring tokens).

ConditionΔPPL vs. Learned95% paired CI
Early Identity+33.19%[29.21%,37.95%]
Late Identity+1.23%[1.02%,1.44%]
Early C4 Mean+0.29%[0.08%,0.53%]

Downstream tasks. For GSM8K, we teacher-force all 1,319 gold solutions (135,596 answer tokens); the metric is conditional answer-token NLL.

ConditionΔNLL vs. Learned95% paired CI
Early Identity+0.024496[+0.021782,+0.027142]
Late Identity+0.000201[−0.000821,+0.001231]

For C-Eval (1,335 questions), we report pooled accuracy.

ConditionAccuracyΔAccuracy vs. Learned (pp)95% paired CI (pp)
Learned88.31%——
Early Identity78.20%−10.11[−12.06,−8.16]
Late Identity88.46%+0.15[−0.60,+0.90]

Long contexts. We test 2K, 4K and 8K contexts on 32 independent C4 documents, scoring the same final 512 tokens with fixed C4-derived mean mixers.

ContextEarly Identity ΔPPL (95% CI)Late Identity ΔPPL (95% CI)Early C4 Mean ΔPPL (95% CI)
2K+26.05% [20.55,32.09]+1.36% [0.80,2.04]−0.08% [−0.59,0.41]
4K+25.54% [20.10,31.60]+2.26% [1.45,3.21]−0.02% [−0.73,0.58]
8K+24.52% [18.54,31.27]+2.25% [1.56,3.00]+0.17% [−0.36,0.73]

Late identity has substantially less impact than early identity, further supporting our finding that residual mixing is primarily concentrated in early layers.

4. Novelty and Scope of the Findings

Previous studies reported globally dominant-stream behavior in smaller HC models. In contrast, we find that routing in DeepSeek-V4-Flash is locally concentrated but reorganized across depth: different streams dominate at different depth ranges, and read/write routing follows distinct patterns. Meanwhile, residual mixing is concentrated in early layers and becomes nearly identity in later layers. Our intervention experiments further connect these structural patterns to functional importance, revealing component- and depth-dependent redundancy rather than global stream collapse.

We will scope our conclusions to the evaluated DeepSeek-V4-Flash checkpoints and metrics, and clarify that the appendix's local gradient analysis is hypothesis-generating.

I thank the authors for the detailed response. The added token-level effective-stream counts, paired confidence intervals, long-context probes, and explicit scoping commitments are helpful. However, several concerns still remain.

Reporting NtokenN_{\mathrm{token}} is useful, and the values of 1.53–2.02 support concentrated routing. However, NtokenN_{\mathrm{token}}, NseqN_{\mathrm{seq}}, and NpooledN_{\mathrm{pooled}} estimate different quantities. The paper should define their averaging order and weighting explicitly, report direct uncertainty comparisons, and provide site- or layer-level distributions. The value 1.53 for write-attention and 1.77 for read-FFN shows that “approximately two streams” is only a coarse aggregate summary. The conclusion should specify which estimator supports this statement and should avoid interpreting NseqN_{\mathrm{seq}} or NpooledN_{\mathrm{pooled}} as the typical number of streams used by an individual token.

Moreover, the paired CIs do not support treating late-mixer replacement as performance-preserving. Late identity increases C4 perplexity by 1.23% with CI [1.02, 1.44], and long-context perplexity by 1.36–2.26% with CIs excluding zero. This is a small but statistically reliable degradation, not equivalence. The claim “little measured benefit” should be replaced by “small but nonzero degradation on the evaluated metrics”; “dispensable” is unsupported. And downstream CIs are provided only for C-Eval and GSM8K NLL, not for the six-task average or all individual tasks. These should be added. The long-context evidence is also limited to 32 documents and the final 512 scored tokens, so conclusions should be scoped accordingly.

Additionally, the depth-resolved, intervention-linked characterization is a useful empirical contribution beyond prior small-scale HC work. However, it is not a mechanistic explanation. The response should state explicitly which findings are truly new—e.g., depth-varying local routing, early-only residual-mixing benefit, and intervention-localized redundancy—versus large-scale confirmations of prior observations.

I therefore maintain Weak Reject.

Reviewer_veWtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines how DeepSeek-V4-Flash uses its mHC residual streams in practice. The authors analyze read/write routing, residual mixing, and inter-stream representation similarity across layers, then use routing sparsification and residual-mixer replacement at inference time to test the functional importance of these components. The mixer interventions help distinguish the contribution of learned site-specific structure from that of token-wise variation. Evaluations using C4 perplexity and downstream benchmarks connect the observed structural patterns to their effects on model performance.

Critical strengths

Replacing early mixers with their site-specific means leaves C4 perplexity and the average downstream score nearly unchanged, whereas replacing them with identity substantially increases perplexity. This contrast helps distinguish the benefit of the learned mixing structure from that of token-wise variation.

Across layers, the dominant stream changes even though routing remains locally concentrated. The paper captures this distinction clearly and is careful not to treat directional differences between stream representations as evidence of functional importance.

Critical weaknesses

GSM8K accuracy drops under several interventions even when the average downstream score barely changes. Without confidence intervals for these differences and a fuller description of the generation settings, it is difficult to assess the reliability and practical significance of these losses.

Only one checkpoint is examined, with diagnostics drawn from short C4 sequences and downstream tests confined to English-language benchmarks. Whether the same routing patterns and limited benefit of token-wise variation in the residual mixers persist on different input distributions or longer contexts remains an open question.

Questions for authors
  1. Could you provide the GSM8K generation settings, answer-extraction procedure, and paired confidence intervals for the accuracy differences from baseline?
  2. Why does applying top-2 routing to the write map HpostH^{\mathrm{post}} cause a much larger drop in GSM8K accuracy than applying it to the read map HpreH^{\mathrm{pre}}?
  3. How sensitive are the “Token mean” results to the choice and number of C4 diagnostic sequences?
  4. Do you have additional evaluations showing whether the findings from residual-mixer replacement hold on other input distributions or longer sequences?
2 comments

We thank the reviewer for the constructive feedback and positive assessment of our mHC residual-stream analysis.

1. GSM8K Evaluation Protocol and Paired Uncertainty

As described in Section 4.1, we evaluate downstream tasks using standard LM Evaluation Harness protocols. We detail the GSM8K settings below.

Generation. We evaluate all 1,319 GSM8K test questions using 5-shot prompting with the plain-text format Question: ...\nAnswer: and greedy decoding (do_sample=false, temperature=0) with a 2,048-token context budget and a 256-token generation limit. Generation stops at the configured stop strings (Question:, </s>, <|im_end|>) or tokenizer EOS.

Answer extraction. We use exact_match,flexible-extract, which selects the last numeric match via the task's regex filter, followed by case-insensitive exact-match scoring after normalization (removing commas, dollar signs, text through ####, and trailing periods). Outputs without numeric matches are marked [invalid] and scored incorrect. The same procedure applies to all interventions.

Paired uncertainty. The original evaluation retained only aggregate scores, precluding full-test paired CIs. We randomly sample 64 of the 1,319 questions and evaluate the baseline, read-top-2, and write-top-2 settings on the same subset using the original 256-token protocol. We report accuracy differences (pp) and 95% paired-bootstrap CIs from 10,000 resamples of shared question IDs.

SettingAccuracy (%)Δ vs. Baseline (pp)95% Paired CI (pp)
Baseline95.310.00—
Read Top-289.06-6.25[-14.06,0.00]
Write Top-262.50-32.81[-45.31,-20.31]

2. Asymmetric Sensitivity of Read and Write Routing

We attribute this asymmetry to the distinct roles of the two maps: the read map aggregates information from residual streams for the current block, whereas the write map distributes block updates back to the streams, shaping subsequent residual states.

To isolate routing effects from generation-history differences, we evaluate all 1,319 GSM8K solutions (135,596 answer tokens) using teacher forcing, conditioning each token on the same ground-truth prefix. We apply top-2 sparsification separately to the read and write maps, leaving other maps learned.

Read Top-2 increases answer-token NLL by 0.0651 (+6.72% PPL), while Write Top-2 causes a much larger increase of 0.2529 (+28.77%). Both increases are supported by paired 95% CIs:

InterventionΔNLL95% Paired CIRelative PPL Change
Read Top-2+0.0651[0.0617,0.0686]+6.72%
Write Top-2+0.2529[0.2440,0.2619]+28.77%

The larger write-top-2 degradation persists under identical ground-truth prefixes.

3. Sensitivity to C4 Diagnostic Sequences

We use an independent pool of 192 C4 documents (2,048 contiguous tokens each). For three sampling seeds (11, 29, 47), we estimate token-mean mixers from nested 8K, 32K, and 128K-token subsets, replacing mixers in layers 0–21. All variants use the same held-out 256-document C4 set (524,032 tokens).

Across all nine settings, PPL increases remain small (0.28–0.42%); all paired 95% CIs exclude zero, with no consistent trend across calibration sizes. This suggests limited sensitivity to calibration sampling, although the replacement is not strictly lossless.

Calibration TokensSeedMean NLLRelative PPL Change95% Paired CI
8K112.185668+0.4153%[0.1356%,0.7362%]
8K292.184375+0.2855%[0.0644%,0.5277%]
8K472.184886+0.3368%[0.1028%,0.5904%]
32K112.185385+0.3868%[0.1321%,0.6724%]
32K292.184306+0.2786%[0.0343%,0.5619%]
32K472.184991+0.3473%[0.1079%,0.6081%]
128K112.184329+0.2809%[0.0385%,0.5504%]
128K292.184859+0.3341%[0.0570%,0.6691%]
128K472.184928+0.3410%[0.1098%,0.5975%]

4. Generalization Across Input Distributions and Longer Contexts

Cross-distribution generalization. We evaluate fixed C4-derived mean mixers on C4, GSM8K, and C-Eval without domain-specific recalibration. C4 mean replacement has limited impact across all three datasets, unlike early identity replacement.

DatasetMetricEarly IdentityC4 Mean (95% CI)
C4ΔPPL (%)+33.19+0.29 [0.08,0.53]
GSM8KΔNLL+0.02450-0.00155 [-0.00294,-0.00014]
C-EvalΔAccuracy (pp)-10.11-0.82 [-1.95,+0.30]

Long-context evaluation. We evaluate 2K, 4K, and 8K contexts on 32 C4 documents, scoring the same final 512 tokens at each length with fixed C4-derived mean mixers.

ContextEarly Identity ΔPPL (95% CI)Late Identity ΔPPL (95% CI)Early C4 Mean ΔPPL (95% CI)
2K+26.05% [20.55,32.09]+1.36% [0.80,2.04]-0.08% [-0.59,0.41]
4K+25.54% [20.10,31.60]+2.26% [1.45,3.21]-0.02% [-0.73,0.58]
8K+24.52% [18.54,31.27]+2.25% [1.56,3.00]+0.17% [-0.36,0.73]

Early identity consistently degrades performance, while C4 mean replacement shows little measured impact across context lengths up to 8K.

Thank you for the additional information, which have addressed some of my concerns. However, I still believe the initial rating reflects the assessment of the paper. I would like to keep my score.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your time and constructive feedback on our submission. As the discussion period is drawing to a close, we would like to gently remind you that we have posted our responses to address all of your comments.

We would greatly appreciate it if you could take a moment to review our rebuttal and let us know if any concerns remain. We are happy to provide any further clarifications before the deadline.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.