acceptodds
Under review as a conference paper at ICLR 2027

The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise.

Abstract

Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their "expert specialization" remain poorly understood. We show that, since MoE routers are linear maps, hidden state similarity is both necessary and sufficient to explain expert usage similarity, where inputs from similar domains receive similar routing decisions, i.e. expert specialization. We confirm this at both token and sequence level across five pre-trained models. We additionally prove that load-balancing loss suppresses shared hidden state directions to maintain routing diversity, which might provide a theoretical explanation for specialization collapse under less diverse data, e.g. small batch. Despite this clean mechanistic account, we find that specialization patterns in pre-trained MoEs resist human interpretation: expert overlap between different models answering the same question is no higher than between entirely different questions (60%); prompt-level routing does not predict rollout-level routing; and deeper layers exhibit near-identical expert activation across semantically unrelated inputs in certain models due to specific output structure. We conclude that, while the efficiency perspective of MoEs is well understood, understanding expert specialization is at least as hard as understanding LLM hidden state geometry, a long-standing open problem in the literature.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture of experts

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_spZ4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies hidden state similarity as a tool to understand expert usage similarity and derive a theorem to interpret their relationship. Yet the authors find it difficult to understand specialization patterns in pre-trained LLMs, which can exhibit different pattterns in prefilling / decoding (and in decoding it means it is hard to understand which expert is used during CoT reasoning for instance). Lastly, they propose a method to mitigate usage similarity, which seems to work in early layers.

Critical strengths

This work provides a detailed analysis of the relationship between hidden state similarity and the expert specialization phenomenon, which is convincing and easy to follow.

Critical weaknesses

The method in practical takeaway does not seem to be very convincing, because starting from layer 10, the similarity is very similar to the one without depth scaling, do you think there might be a better method that can make similarity low even in deeper layers, or is that impossible, if impossible, could you explain why.

Questions for authors
  • Line 044, there are many alternate router choices other than the linear projection, you could say instead by "arguably, the linear projection is one of the most popular choices for the router", then continue your argument
  • Section 4.1, it might be better to move the ||P||_2 ||hi - hj||2 up to before section 4.1 so it would easier to follow. How would you estimate r in real data?
  • Figure 1, it might be more convincing if more sequences could be used.
  • Figure 2, It is unclear to me why hamming similarity is used on the right, while cosine similarity is used in the middle. Also it is unclear to me what energy means in the main text, and why Muon is used in the bottom plot. Better explanation could make clarity better in this part.
  • How does "the router is not a random matrix" support the claim in Section 4 that expert specialization arises from hidden state similarity?
  • Figure 4 is a bit hard to read, it might be better to plot in a way that x-axis is layer, y-axis is the activated frequency difference, then can see the difference grows with layer.
  • Line 364-371, diferent models would have different activated experts might not be surprising, because they are in different archietectures and are trained in potentially different ways. Above are mostly writing concerns, I would change from "Weak Accept" to "Accept" if the authors could address my main question in the "Critical weakness" part.
6 comments
AuthorsOpenReview ↗

Author response (1/2)

We would like to thank the reviewer for the detailed and careful review!

Please see below for our responses to the weaknesses and questions. All revised sentences in the manuscript are highlighted in red

Alternative methods for reducing hidden state collapse

Yes, there are alternative methods to further reduce the hidden state similarity

  • Increase the strength of depth scaling: For a L−L-layer model, depth scaling weights the contribution of each by c=α/Lc=\alpha / \sqrt{L}, reducing α\alpha can further reduce the similiarity.
  • Initialize the attention's value or output projection weights as 0, which eliminates the major source of similarity injection at each layer.

We provide illustrations for the two approaches in the updated Figure 8 in the main text of the updated manuscript; both approaches further reduce hidden state similarity and enhance routing balance at initialization.

It is worth noting that the zero attention weight trick is only effective at initialization time since the attention weights CANNOT all be zero after training. The residual scaling technique could remain effective as it works by downweighting the correlation injection at each layer (although a theoretical derivation is no longer possible, as the independence assumption no longer holds).

When moving away from the initialization, we also observed that the model trained with Muon instead of AdamW tends to have less correlated hidden states, as shown in the left column in Figure 2, as well as the left column in Figure 17 (b) v.s. Figure 17 (c), where an increasing energy in the first direction (0:10:1) with depth indicates stronger shared direction in the data and hence more correlation.

AuthorsOpenReview ↗

Author response (2/2)

Writing concerns

Line 044, there are many alternate router choices other than the linear projection, you could say instead by "arguably, the linear projection is one of the most popular choices for the router", then continue your argument

Good point! We have revised that sentence.

Section 4.1, it might be better to move the ||P||_2 ||hi - hj||2 up to before section 4.1 so it would easier to follow. How would you estimate r in real data?

To select r, we compute the SVD of the input hidden-state matrix (H) and choose the smallest r such that the leading r singular directions capture a prescribed fraction (e.g., 90%) of the total energy, measured by the sum of squared singular values: ∑k=1rσk2∑k=1mσk2≥0.9.\frac{\sum_{k=1}^{r}\sigma_k^2}{\sum_{k=1}^{m}\sigma_k^2}\geq 0.9.

Figure 1, it might be more convincing if more sequences could be used.

Here, each dot in the scatter plot denotes a token and 16 sequences already included ~20K tokens / dots. Including more sequences would significantly slow down the visualization and makes the figure harder to read (it will look like a puddle of blue paint).

Figure 2, It is unclear to me why hamming similarity is used on the right, while cosine similarity is used in the middle. Also it is unclear to me what energy means in the main text, and why Muon is used in the bottom plot. Better explanation could make clarity better in this part.

Cosine and Hamming measures similarity on different quantities:

  • Cosine similarity measures the similarity between two continuous routing logits vectors.
  • Hamming similarity measures the similarity of the actual expert activation after top-K, i.e. a discrete binary vector. We use cosine similarity in the middle to reflect its relationship with the hidden state's trend. We use hamming similarity on the right to study the actual expert usage diversity.

The energy in the left column refers to the normalized sum of squared singular values (Equation 9 in appendix A), the retained energy on the right refers to magnitude of router's projection on the data's first singular direction, i.e. how sensitive is the routing against certain direction in the data (Equation 10 in appendix A)

The reason Muon is studied here is that different optimizers show different energy distributions in the hidden states (left column): Hidden states trained under AdamW show an increased energy in 0:1, i.e., a stronger shared direction, more correlation and more hidden state collapse. Models trained Muon shows much lower energy from the first direction and less hidden state collapse with respect to depth.

We include AdamW v.s. Muon comparison here as we hypothesis using Muon can significantly alleviating the routing imbalance caused by hidden state collpase, and hence the pre-training can rely purely on auxiliary loss free load balancing for minor correction.

How does "the router is not a random matrix" support the claim in Section 4 that expert specialization arises from hidden state similarity?

The implication is illustrated in the "router-data alignment" term (note that a random matrix would have a non zero value): The router data alignment determines how sensitive the logits are to the hidden state distance.

There are two practical impacts:

  • As we argued in Proposition 2, if load balancing loss is set too large, then the alignment will be very low; therefore, even very large hidden state distance would induce low logits distance
  • Additionally, not related to expert specialization, when the input is out of distribution, it will have low alignment with the router weights, and hence lower logit magnitude and lower routing confidence value; which enables us to use the router as a OOD detection tool (Figure 3).

Figure 4 is a bit hard to read, it might be better to plot in a way that x-axis is layer, y-axis is the activated frequency difference, then can see the difference grows with layer.

We agree! However, we plot it this way to avoid additional aggregation over experts (we already aggregate over all tokens) such that the differences can be illustrated directly. We find an aggregated "similarity number" (e.g. the cos similarity we used in Figure 11) less informative than a direct visualization of expert usage.

Line 364-371, diferent models would have different activated experts might not be surprising, because they are in different archietectures and are trained in potentially different ways.

Agreed! But the main takeaway here is that it's difficult to reason about the expert activation pattern:

  • These rollouts are all solutions for the same set of math questions, so they belong to the same domain and could activate similiar experts.
  • On the other hand, for very long rollouts from models trained differently and with different architectures, the solution string could differ significantly distribution-wise, hence activating different experts. Simply relying on the "metadata" of math corpus is not sufficient to understand the behavior.

Dear Reviewer spZ4,

Thank you for your review. The authors have posted their response. Please add a brief follow-up indicating whether your concerns have been addressed and noting any remaining questions.

The discussion phase closes on November 18 at 11:59 pm AoE.

Thank you for your attention and contribution.

Best regards,

Area Chair

Reviewer_xGenOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether expert routing behavior in MoE models provides reliable evidence of domain specialization. Through theoretical analysis and experiments across multiple models, the authors examine how hidden-state geometry, context, load balancing, and differences between prefilling and decoding affect expert selection. The main contribution is to demonstrate that expert usage frequencies and domain-dependent routing patterns alone are insufficient to establish domain expertise.

Critical strengths
  1. The paper draws a valuable distinction between expert selection and functional expertise, challenging interpretations of domain specialization based primarily on routing statistics.
  2. The experiments examine routing behavior across multiple models and settings. The expert-pruning analysis further demonstrates the practical consequences of differences between prefilling and decoding, offering useful insights for expert analysis and pruning.
Critical weaknesses

The abstract describes hidden-state similarity as “necessary and sufficient” to explain similar expert usage, but the theoretical result appears to support a narrower claim. It bounds how much routing scores can differ when hidden states change. Different hidden states may still produce the same routing scores, while small score changes can alter which experts are selected when their scores are close.

Questions for authors
  1. Could the authors clarify the variables and assumptions underlying the “necessary and sufficient” claim? The theoretical result in Section 4.1 primarily concerns the relationship between hidden states and routing logits. Extending this result to top-k expert selection requires accounting for selection margins. Please provide the relevant conditions or revise the abstract and associated conclusions.

  2. The manuscript would benefit from further language polishing. Frequent semicolons and lengthy compound sentences interrupt the flow of the exposition and make it difficult to distinguish observations, mechanistic explanations, and hypotheses. Shorter sentences and more direct statements of the conclusions supported by each experiment would improve readability.

3 comments
AuthorsOpenReview ↗

Author response

We would like to thank the reviewer for the careful and insightful comments!

Please see below for our responses to the weaknesses and questions. All changes in the manuscript are highlighted in red.

Necessary and sufficient conditions

Wording in the abstract

We have changed the wording in the abstract as "hidden states plays a critical role in explaining expert usage similarity". Implying that necessity is not our intention, as we highlighted in multiple places in the draft:

Line 178, the "Triangle shape of hidden states vs logit distances" paragraph.

Line 344:

... dissimilar sequences can activate either similar or different experts, a many-to-one relationship that already hints at interpretive difficulty...

Our original intention is to highlight that it is very necessary to study hidden state similarity in order to understand expert specialization. Regardless, we have updated the wording.

Selection margin bridges hidden state similarity and routing decision similarity

We also agree that a small logit distance does not by itself guarantee similar routing decisions. The guarantee depends on the selection margin, denotes as $\Delta_k(g) $:

Δk(g)=g(k)−g(k+1),\Delta_k(g) = g_{(k)} - g_{(k+1)},

where g=Phig = P h_i is the vector of router logits and g(k)g_{(k)} denotes its kk-th largest entry.

Let g′=Phjg' = P h_j. If

∥g−g′∥2<Δk(g)2,\lVert g - g' \rVert_2 < \frac{\Delta_k(g)}{\sqrt{2}},

then TopK(g)=TopK(g′)\mathrm{TopK}(g) = \mathrm{TopK}(g'), where TopK\mathrm{TopK} denotes the set of selected experts.

To see why, let δ=g′−g\delta = g' - g. For any expert ee selected under gg and any expert e′e' unselected under gg, we have ge−ge′≥Δk(g)g_e - g_{e'} \geq \Delta_k(g). By Cauchy–Schwarz, ∣δe−δe′∣≤2,∥δ∥2|\delta_e - \delta_{e'}| \leq \sqrt{2}\\,\lVert \delta \rVert_2. Therefore,

ge′−ge′′≥Δk(g)−∣δe−δe′∣≥Δk(g)−2,∥δ∥2>0.g'_e - g'_{e'} \geq \Delta_k(g) - |\delta_e - \delta_{e'}| \geq \Delta_k(g) - \sqrt{2}\\,\lVert \delta \rVert_2 > 0.

Thus, every selected expert still outranks every unselected expert. ■\blacksquare

Combining this result with Proposition 1, tokens ii and jj select the same experts whenever

∥PΠr∥2,∥hi−hj∥Πr+∥P(I−Πr)(hi−hj)∥2<Δk(Phi)2.\lVert P \Pi_r \rVert_2 \\, \lVert h_i - h_j \rVert_{\Pi_r} + \lVert P(I - \Pi_r)(h_i - h_j) \rVert_2 < \frac{\Delta_k(P h_i)}{\sqrt{2}}.

In other words, similar hidden states activate the same experts when the resulting bound on logit distance is smaller than the selection margin divided by 2\sqrt{2}.

Language polishing

We have revised the writing in several places; please see the highlighted red sentences:

Last paragraph in the introduction section.

Line 363, 375, 413 and line 425 in Sec 4.4

Line 466 and 475 in Sec 5.

The authors’ response resolves most of my concerns by revising the “necessary and sufficient” claim and providing a selection-margin condition linking hidden-state similarity to identical top-k expert selection.

Thus, I maintain my recommendation.

AuthorsOpenReview ↗

Thank you for the review!

Dear reviewer xGen

We would like to thank you again for the suggestions and comments. We believe the revised writing and the extra selection margin make the paper much clearer!

Thanks

Authors

Reviewer_4kJUOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that MoE expert specialization reflects hidden-state geometry, since a linear router must send similar states to similar experts. Measurements on several pretrained MoEs support this at token and sequence level, while finer comparisons show that routing does not match human domain labels.

Critical strengths

The geometric account is consistent across five pretrained MoEs at both token and sequence level, and it explains why inputs from different datasets activate different experts without assuming domain modules.

Separating prefill from generation is useful.

Critical weaknesses

The abstract says hidden-state similarity is necessary and sufficient for similar expert usage. Proposition 1 is only an upper bound. The triangle in Figure 1 shows that dissimilar states can still activate similar experts, so necessity does not hold.

Proposition 2 is derived only for a small-logit regime and a squared deviation of mean softmax probabilities from uniformity, rather than the hard/soft Switch-style objective used by Qiu et al. Figure 12 further uses a coefficient of 1.0 for only 2,000 steps and measures singular-direction alignment, not cross-topic specialization. The result therefore suggests a plausible mechanism, but does not establish that it explains Qiu et al.'s small-batch specialization collapse.

Figure 6 does not match traces on length, template, or lexical overlap, so the 60% overlap may reflect surface-form differences between generators. Figure 7 uses three hand-picked pairs on one model. Figure 2 changes the optimizer, the loss, and the model together, so the effect cannot be attributed to the auxiliary loss alone.

The practical section mostly synthesizes known recipes: depth scaling and zero-initialized attention are already used in large-scale training, while explicit supervision for modular experts is supported by prior citations rather than tested here. Figure 8 provides incremental evidence connecting rank-collapse mitigation to routing balance at initialization, but not a new training method.

Questions for authors
  1. In Figure 6, what are the length distributions, lexical overlap, and pooled hidden-state cosine similarity for the three pair types? If those track the same 60% versus 80–85% gap, the Jaccard result follows from surface form rather than showing that problem identity is irrelevant.

  2. Does the small-logit assumption in Lemma 2 hold on the evaluated models? If the loss is replaced by the Switch or DeepSeek auxiliary loss, does a near-minimizer still have to ignore the shared direction?

  3. Why is the auxiliary coefficient set to 1.0 in Figure 12? At a realistic coefficient such as 10^-2 or 10^-3, and with cross-domain expert overlap measured directly, does the reversed-U alignment remain, and does specialization actually disappear?

  4. Which five models, layers, and routers are in Figure 1? Please state the layer index, top-k, whether routing uses softmax or sigmoid, and whether shared experts are included.

6 comments
AuthorsOpenReview ↗

Author response (1/3)

We would like to thank the reviewer for the great and insightful comments. We find these questions very helpful for improving our work and clarifying our claims.

Necessity of proposition 1

We have changed the wording in the abstract to "hidden state similarity plays a critical role...", implying that the necessity is not our intention, as we highlighted in multiple places in the draft:

Line 178, the "Triangle shape of hidden states vs logit distances" paragraph.

Line 344:

... dissimilar sequences can activate either similar or different experts, a many-to-one relationship that already hints at interpretive difficulty...

Our original intention is to highlight that it is very necessary to study hidden state similarity in order to understand expert specialization. Regardless, we have updated the wording.

AuthorsOpenReview ↗

Author response (2/3)

Proposition 2

Connection with observations in Qiu 2025

Thanks for the comments. We agree that our study considers a setting simpler than the large-scale pre-training settings in Qiu 2025 and our current evidence only supports a potential mechanism rather than a complete explanation. We have revised the corresponding wording in the paper to "We believe this result can provide a possible mechanism for the specialization collapse in Qiu 2025" (highlighted in red)

Switch-style objective and Small logits

The small logits assumption is necessary for linearizing the softmax in order to derive a closed-form solution. The assumption holds for the top-K logits, but it does not necessarily hold for all logits. We have revised the paper to acknowledge the limitation.

The switch style loss does not admit a closed-form solution, therefore, we include a simulated study where we also remove the linearizable softmax assumption. Specifically, we consider inputs

\mu= u,\qquad \epsilon_i=\sigma(\xi_i-\bar{\xi}),\qquad \bar{\xi}=\frac{1}{N}\sum_{i=1}^{N}\xi_i,$$ where $D=1024$, $\|u\|_2=1$ and the uncentered noise vectors $\xi_i$ are sampled uniformly from the unit sphere in $\mathbb{R}^{D}$. We vary $\sigma\in\{0.5,1,2\}$ to adjust the strength of the shared direction. We then minimize the router on the input using switch-style load-balancing loss. Our results **confirm the intuition behind Proposition 2** (Figure 33 in the updated manuscript): The router *gradually suppresses the shared direction during loss minimization*. However, unlike Proposition 2, it does not entirely remove the shared direction; The router re-purposes the shared direction into **a small bias vector** that eliminates the tiny imbalance caused by finite-sample error from $\epsilon$s, which cannot be summed out as in Lemma 2 due to the lack of a linearized form. The magnitude of the bias vector depends on $\sigma$: The larger the $\sigma$ is, the bigger the magnitude is. # Smaller auxiliary coefficient, cross-domain expert overlap and alignment > Why is the auxiliary coefficient set to 1.0 in Figure 12? At a realistic coefficient such as 10^-2 or 10^-3, and with cross-domain expert overlap measured directly, does the reversed-U alignment remain, and does specialization actually disappear? The main purpose of Figure 12 is to illustrate the interaction between batch size, load-balancing loss and data-router alignment. We have **additionally performed a new experiment to answer these three questions**, where we train a 500M MoE under an auxiliary loss coefficient of 0.001 and 0.1 with a data mixture composed of 50\% wikitext and 50\% math corpus from Nemotron math for 1,000 steps. Each step is composed of 8 microbatch steps. We then vary the way auxiliary loss is computed: Window size of 1 v.s. 8, i.e. using the statistics over the microbatch v.s. the minibatch. After the training, we study the relationship between router-data alignment v.s. expert usage overlap for the two domains. Instead of looking at per-direction alignment, we now aggregate over all directions for measuring *router–data alignment*. Denote the centered input covariance as $C=\sum_{i=1}^d \lambda_i v_i v_i^\top$ and the router weights as $W_c=W-\mathbf{1}_E\bar w^\top$, centered across experts to remove a common logit shift. Our metric is $$A = d\\,\frac{\operatorname{tr}(W_c C W_c^\top)}{\lVert W_c\rVert_F^2\\,\operatorname{tr}(C)} = d\sum_{i=1}^d p_i q_i, \qquad p_i = \frac{\lambda_i}{\operatorname{tr}(C)}, \quad q_i = \frac{\lVert W_c v_i\rVert_2^2}{\lVert W_c\rVert_F^2}.$$ Here $p_i$ and $q_i$ are the fractions of input variance and squared router weight norm along $v_i$, respectively. An isotropically oriented router has $\mathbb{E}[A]=1$, and larger / smaller values indicate greater alignment with high/low variance directions, respectively. Results are shown in Figure 34 (page 39) in the updated manuscript, where we show expert overlap v.s. alignment ($A$) across settings. Two main observations stand out: - For all top 9 (out of 12) layers, **expert overlap reduces as router-data alignment increases.** - A *larger* auxiliary loss coefficient, as well as a *smaller batch size* for computing balancing statistics, reduces router data alignment. *Both observations support the claim we attempted to make with Figure 12.*
AuthorsOpenReview ↗

Author response (3/3)

Other metrics for Figure 6

We evaluated the three metrics on gpt-oss

Length distributions: All rollouts collected are truncated to have 5,000 tokens at max; In addition, the similarity metrics are computed by aggregation over all tokens. Therefore, length is not a factor.

Lexical overlap: Lexical overlap shows similiar order compared with expert usage similarity

Different questions, same modelSame question, different modelsSame question/model, different answers
0.2150.2270.449

Pooled hidden state similarity: Hidden state similarity v.s. layer shows similiar pattern and shape compared with expert overlap (Figure 32 in the appendix of the updated manuscript), which confirms the observation in Figure 5: Expert usage similarity originates from hidden state similarity.

Both results suggest that in the Math question experiment, the difference in expert usage originates from lexical difference, which got propagated into the representation space and the router, confirming the claim of our work.

Results in Figure 2 and Figure 7

Figure 7 uses three hand-picked pairs on one model.

We use hand-picked examples here to provide "counterexamples" to demonstrate the unpredictability of prefill v.s. generation time expert usage. Gpt-oss-20b's experiments in Figure 9 and 10 confirm a similiar claim but using entirely different behavior unique to that model alone. Overall, we believe the behavior varies by model (e.g. reasoning v.s. non-reasoning ones) and by prompts; therefore, there is no one single conclusion that can be drawn here other than "the pattern is hard to predict"

Figure 2 changes the optimizer, the loss, and the model together, so the effect cannot be attributed to the auxiliary loss alone.

In Figure 20 and 21 in the appendix, we compare two models with identical architecture, but trained under different optimizer and load-balancing regimes. The one trained with Muon shows significantly less hidden state collapse compared with the one trained with AdamW (left columns in Figure 17 c v.s. Figure 17 b). And we hypothesize that the healthier hidden state reduces the stress for load balancing such that the auxiliary free bias mechanism alone is sufficient.

We cannot ablate the choice of optimizers as we are unable to find any open-source models trained w/o auxiliary loss + AdamW at the time of writing; We believe this highlights the entanglement between hidden state dynamics and load balancing, the central claim of our paper.

Practical takeaway

Our work does not directly propose any "new approaches"; our contribution lies in connecting two separate dots in pre-training research: efficient use of depth and load balancing.

At initialization, depth scaling and zero-initialized attention are old ideas that have been widely studied in signal propagation literature for improving effective use of depth; Our work novely shows that they also enable effective use of width, i.e. non-collapsed routing at initialization.

Our hidden-state-driven-routing perspective also suggests during training, techniques that can improve hidden state collapse e.g. using Muon over AdamW / residual scaling / residual attention, can also substantially reduce the pressure of load balancing.

Other questions

Which five models, layers, and routers are in Figure 1? Please state the layer index, top-k, whether routing uses softmax or sigmoid, and whether shared experts are included.

The model names are stated at the top of each subfigure. The layer is picked as the middle layer of each model (similiar observation also holds for other layers, as shown in Fig 22 - 24). The model specific MoE design are summarized below

ModelTop-k routed expertsRouter scoringShared experts
ERNIE-4.5-21B-A3B-Base6 of 64Softmax before top-k2
gpt-oss-20b4 of 32Softmax over selected logits0
DeepSeek-V2-Lite6 of 64Softmax before top-k2
Moonlight-16B-A3B6 of 64Sigmoid before top-k2
Trinity-Mini-Base8 of 128Sigmoid before top-k1

We have also included the summarization in the beginning of the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.