acceptodds
Under review as a conference paper at ICLR 2027

CoFFiE: Consensus-First Fine-Tuning for MoE

Abstract

Mixture-of-Experts (MoE) models are increasingly used as base models for downstream tasks, but how best to fine-tune them remains an open question. Full fine-tuning stores optimiser state for every expert, even though only a few are active for each token and each expert sees only a small fraction of the fine-tuning data. Parameter-efficient methods reduce this cost, but most inherit designs from dense models: each expert or adapter computes its update independently, whereas the output of an MoE layer depends on how its active experts work together. We propose CoFFiE (Consensus-First Fine-tuning for Mixture-of-Experts), which keeps the experts and the router frozen and adapts this coordination instead. A small shared network refines each expert's output using its own state and the route-weighted consensus of the active experts. Because refining the experts also changes this consensus, we define the refined outputs implicitly, as their joint fixed point. A few iterations suffice in practice, and training memory does not grow with the iteration count. We further extend the method by propagating consensus information across layers, which also improves accuracy significantly. On OLMoE-1B-7B, Mixtral-8x7B and Qwen3-30B-A3B, CoFFiE outperforms the strongest MoE-specific PEFT baselines with the same or fewer trainable parameters, by up to 3.3 points on commonsense and arithmetic reasoning benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, parameter-efficient fine-tuning, deep equilibrium models, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_M5mxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoFFiE, a parameter-efficient fine-tuning (PEFT) method for Mixture-of-Experts (MoE) models. While existing MoE PEFT methods typically adapt selected experts independently, CoFFiE trains a small shared network to jointly refine each expert's output by conditioning it on information from other co-activated experts. To ensure consistency between the refined expert outputs and their consensus, the authors formulate the refinement process as a coupled fixed-point problem. Experiments on three MoE models demonstrate improved performance with a small number of trainable parameters.

Critical strengths

1. Parameter-efficient and practical design.

CoFFiE employs a lightweight refinement network shared across active experts, allowing cross-expert information to be incorporated without increasing the number of trainable parameters proportionally to the total number of experts. Additionally, the proposed training procedure avoids backpropagating through the forward solver iterations, keeping the backward memory cost independent of the number of iterations. These properties make the approach practically appealing for fine-tuning large MoE models.

2. Evaluation across multiple MoE architectures.

The paper evaluates CoFFiE on three MoE backbones, demonstrating consistent improvements over the evaluated baselines on commonsense and arithmetic reasoning benchmarks. The paper also provides implementation details, code repository references, and ablation studies, which are valuable for reproducibility and understanding the proposed method.

Critical weaknesses

1. Further justification for consensus-conditioned and fixed-point refinement.

The paper motivates expert coordination by the lack of explicit interaction among co-activated experts, but provides limited analysis of why independent adaptation is problematic in practice, such as conflicting corrections or redundant updates.

Tables 3 and 7 support the usefulness of consensus-conditioned feedback, but do not clearly establish the additional benefit of fixed-point consistency. Accuracy is similar for n=1, 3, and 5, and the cross-layer variant performs well even at n=0. While the residual and Jacobian analyses support the solver’s numerical behavior, it remains unclear whether closer approximation to an equilibrium improves downstream performance. A comparison with finite-step refinement trained through explicit unrolling, under matched parameter and computational budgets, would help clarify this contribution.

2. Clarification of baseline selection and statistical reliability.

The experimental results across three MoE backbones are encouraging. The paper states that the baselines are re-evaluated under the same training and evaluation settings using the hyperparameters from Liu et al. (2024). However, it would be helpful to provide more details on how the specific baseline configurations were selected for each backbone. In particular, the evaluated PERFT variants and their configurations differ considerably across models, making it difficult to understand the selection criteria. Clarifying whether these configurations were selected based on prior results, parameter budget constraints, or additional validation would help establish the fairness of the comparisons.

Furthermore, Table 1 reports single-seed results. Although additional multi-seed results are provided for CoFFiE on OLMoE, corresponding statistical comparisons with the baselines are limited. Providing multi-seed results for the strongest baselines would strengthen the reliability of the reported improvements.

Questions for authors

1. Necessity of the fixed-point formulation.

The ablation results suggest that introducing feedback can improve performance, but they do not clearly demonstrate that multiple iterations or convergence toward a fixed point are necessary. In particular, n=1, 3, and 5 achieve statistically indistinguishable results, and the cross-layer variant already performs well with n=0. Could the authors clarify what benefit is uniquely attributable to the fixed-point formulation, beyond single-step consensus-conditioned refinement? Comparisons with simpler iterative refinement methods under matched computational budgets would be particularly helpful.

2. Baseline selection and statistical reliability.

Could the authors clarify how baseline configurations were selected for each backbone and whether comparable hyperparameter search budgets were used? In addition, could the authors provide multi-seed results for the strongest baselines, particularly for the comparisons in Table 1, to establish whether the reported improvements are robust?


Minor Comments

  1. Notation clarity (Section 3.2): The notation (E^\star) is introduced implicitly through the fixed-point equations. Explicitly defining E⋆E^\star before Eq. (3) would improve readability and make the formulation easier to follow.
  2. Inconsistent code repository URLs: The repository URLs in the Abstract and the Reproducibility Statement are inconsistent. Furthermore, the URL in the Reproducibility Statement appears to be inaccessible. Please verify the links and provide a consistent, accessible repository URL.
3 comments
AuthorsOpenReview ↗

Response to Reviewer M5mx

W1 / Q1: (a) Why independent adaptation is problematic; (b) what benefit is unique to the fixed-point formulation beyond single-step consensus refinement; comparison with finite-step refinement trained by explicit unrolling at matched parameters and compute.

We thank the reviewer for these questions.

  1. (a) Why independent adaptation is a limitation. In PERFT, every adapter computes its correction from the layer input alone, in parallel to the frozen experts. PERFT-E gives each expert its own adapter, so a fixed budget is split across all experts and each piece is trained only on the tokens routed to its expert. As a result, each adapter is small and learns from only a fraction of the data. PERFT-R instead adds a separate set of adapters with a new router, and one has to choose how many adapters to add, how many to activate per token and their rank. CoFFiE is simpler to use, since it adds a single operator per layer whose only size parameter is its width, and we use the same structure on all three backbones.
  2. (b) Why iteration. Each expert is corrected using the consensus of the active experts, but this consensus is itself computed from the corrected experts. Correcting the experts changes the consensus, and the new consensus changes the corrections. The refined outputs are therefore defined by a fixed-point equation (Eq. 3, Section 3.2), and iteration is how we solve it. Empirically, iteration improves both versions of the method (General Response A).
  3. Number of iterations. As noted in the paper (Section 4.2), accuracy is similar for n=1, 3 and 5, so a single iteration already gives most of the benefit. We use n=3 because most of our experiments had been run with it, and we kept it for consistency.
  4. Comparison with explicit unrolling. As the reviewer suggested, we trained the same model with gradients passed through all iterations. Unrolling is about 1.6 points more accurate, but it needs about twice the training memory and about 21% more time per step. The one-step gradient is therefore a trade-off between accuracy and training cost (Appendix O).

W2 / Q2: Baseline selection criteria and multi-seed results for the strongest baselines.

We thank the reviewer for these questions.

  1. How the configurations were chosen. We took the best PERFT-R and PERFT-E configurations in the main table of the PERFT paper, which its authors had already optimised, and set the width of our operator so that our parameter count matches PERFT-R. On Mixtral we likewise use the main-table configuration, Top2/2 with LoRA8. Qwen3 is not covered by the PERFT paper, so there we keep the Top8/8 structure and set the LoRA rank to 5 to match our parameter count.
  2. Other configurations. We noticed that the appendix of the PERFT paper lists a few configurations of similar size with higher reported accuracy, and we were curious about this as well. We therefore ran two further configurations ourselves, Top4/4 with LoRA32 and Top4/8 with LoRA16. They reach 69.00 and 68.37, and CoFFiE remains clearly above both.
  3. Reproduction and multiple seeds. With the original configuration we reproduce PERFT-R on OLMoE at 68.86, matching the reported 68.81. Over three seeds PERFT-R reaches 69.04 ± 0.60. These results are in Appendix P.

Minor comments. E* denotes the coupled fixed point defined by Eq. (3), that is, the refined expert outputs. The anonymous repository link in the abstract is the correct one, and we have confirmed that it is accessible.

We appreciate the reviewer's suggestions, in particular the comparison with explicit unrolling, which made the trade-off in our training procedure clearer. We remain available for further discussion, and we hope the reviewer will take the additional evidence into account in the evaluation.

We thank the reviewer again for the review. We would be grateful to know whether our response addresses the reviewer's concerns, and we are glad to clarify anything further.

While answering a follow-up question from Reviewer R16D, we obtained a result that also bears on the reviewer's Q2 about hyperparameter search, and we would like to share it here. PERFT-R does not benefit from a larger learning rate (68.86 at its original 1e-5, against 37.13 at 5e-4), whereas Table 1 reports CoFFiE at 2e-4, not at its best learning rate of 5e-4. We would be grateful to know whether our response addresses the reviewer's concerns, and we are glad to clarify anything further.

Reviewer_VvfLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoFFiE is a PEFT method for pretrained MoE LLMs that keeps the backbone, router, and all experts frozen. Each MoE layer gets one small bottleneck network TθT_\theta, shared by the active experts, which refines each active expert’s output using that output and the route-weighted consensus cc of the active experts.

Critical strengths
  1. A simple, MoE-specific design with careful parameter matching: One operator per layer serves all active experts, so the number of trainable parameters does not grow with the number of experts. Per-expert methods such as PERFT-E cannot even be built at this budget on Qwen3-30B-A3B. The expert-only control widened to h′ = 1.5h shows that the consensus input helps at equal capacity.
  2. Transparent analysis of the mechanism: The paper reports the contraction constant of the trained operator (κ = 2.3 to 5.9), the local spectral radius (one layer at ρ(J) = 1.002), and the sensitivity to the sign of α₀. This lets the reader check the claims instead of taking them on trust.
Critical weaknesses
  1. The gains do not come from the fixed point: The title, the method section and Proposition 1 present CoFFiE as a coupled fixed-point (DEQ) problem, but the ablations point somewhere else. The in-layer consensus operator with feedback reaches 68.06, below the PERFT-R baseline (68.81, Table 1). The improvement over PERFT-R comes from the cross-layer state, and with that state, n = 0 (no fixed-point iteration at all) reaches 69.01, higher than the default n = 3 at 68.78 (Appendix I, Table 3). The paper itself notes that accuracy "no longer depends on the number of solver steps" (Section 4.3). The cross-layer gain also appears only for negative α₀: the trained α drifts to about −0.8 to −1 without settling, so |1−α| is about 1.6 to 1.9 and the layer amplifies its own consensus while subtracting the previous layer's. After training, the condition of Proposition 1 fails in every layer (κ > 1). The best configuration is therefore essentially a feed-forward adapter that reads the current and previous layers' mixed expert outputs with a learned signed coefficient. That is simpler than the mechanism the paper motivates, and it is not explained, so the paper should either be reframed around it or explain why it works.

  2. Missing strong baselines: The comparison is only against attention-only LoRA and the PERFT family. Standard baselines are missing: LoRA or DoRA on all linear layers including the expert FFNs, full fine-tuning, and ESFT, which is discussed in related work but not run. Also missing are MoE SFT methods that, like CoFFiE, give the active experts a persistent shared pathway. For example, [3] adds always-active, per-token-gated condenser experts to address the same problem described in the introduction (each expert sees only a fraction of the data), and DenseMixer [4] is another option. The gains over PERFT-R are about +0.8 on Qwen3-30B-A3B and +1.24 on OLMoE. On the arithmetic suite, OLMoE stays below 10% on GSM8K for every method and several test sets are small, so these margins are hard to judge without stronger baselines and harder generative tasks (e.g. MATH, code or instruction following on Qwen3).

  3. Missing related work: [3] is the closest prior work on consolidating information across sparsely activated experts during fine-tuning and should be discussed. Sparse PEFT methods evaluated under the same Commonsense170K/Math protocol, such as SMT [5], would also be a useful non-low-rank baseline. On the baseline side, DoRA [1] on all linear layers is natural additions, and [2] shows that LoRA can fall well short of full fine-tuning on math and code, which is one more reason to report full fine-tuning here. References:

[1] DoRA: Weight-Decomposed Low-Rank Adaptation. ICML 2024.

[2] LoRA Learns Less and Forgets Less. TMLR 2024.

[3] Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning. COLM 2026.

[5] SMT: Fine-Tuning Large Language Models with Sparse Matrices. ICLR 2025.

Questions for authors
  1. If the cross-layer variant at n = 0 matches or beats n = 3, what does the fixed-point formulation add to the final method? Would a plain cross-layer adapter without consensus iteration be a fairer description, and the right ablation row for Table 1? (Clarification; this could change my rating.)

  2. Why does α need to be negative? With |1−α| around 1.8, the operator receives an extrapolated consensus. Does a version that takes [Eᵢ; cℓ; mℓ₋₁] as input with free weights give the same gain?

  3. How does CoFFiE compare with LoRA on all linear layers including experts, with full fine-tuning, and with ESFT or condenser-expert SFT on Qwen3-30B-A3B at a comparable budget?

  4. What is the inference overhead? Unlike LoRA, Tθ cannot be merged into the weights and runs n+1 times per token in each MoE layer. Could the authors report decode latency and throughput relative to the frozen model and PERFT-R?

4 comments
AuthorsOpenReview ↗

Response to Reviewer VvfL

Q1: If cross-layer n=0 matches or beats n=3, is a plain cross-layer adapter without iteration a fairer description, and the right ablation row for Table 1?

We thank the reviewer for suggesting this ablation, which we have now run. At a larger learning rate, the cross-layer version without iteration (n=0) is clearly below the version with iteration (General Response A). Iteration also helps in the free-weight form suggested in Q2 (General Response B). The n=0 row is reported as an ablation in Appendix L.

On the two related points in the review:

  1. In-layer operator vs. PERFT-R. At the same 8.65M budget as PERFT-R, the in-layer operator reaches 69.18, above PERFT-R at 68.81 (Table 9). The 68.06 figure quoted in the review belongs to a smaller operator with 6.29M parameters.
  2. Convergence. The iteration remains locally convergent after training, with a spectral radius below 1 in 15 of 16 layers for the in-layer version and in all 16 layers for the cross-layer version. Proposition 1 gives a sufficient condition for convergence, not a necessary one.

Q2: Why does α need to be negative? Does a version with [E_i; c^ℓ; m^(ℓ−1)] as input and free weights give the same gain?

We thank the reviewer for this suggestion, which we have implemented. The free-weight version reaches the same accuracy as the α version, and iteration improves it as well, from 71.68 at n=0 to 72.57 at n=3 (General Response B).

The sign of α is chosen by training rather than imposed. α moves in the negative direction from every initialisation we tried, and a negative initialisation only shortens the path (General Response B).

Q3 / W2: Comparison with all-linear LoRA, full fine-tuning, ESFT and condenser experts at a comparable budget; harder generative tasks.

We sincerely thank the reviewer for these constructive suggestions; they are valuable reference points for our work.

  1. Harder generative tasks. On MATH-500 with Qwen3, CoFFiE reaches 56.2% and PERFT-R 54.6% (+1.6), so CoFFiE is also ahead on this harder task. We also note that on Mixtral, where GSM8K is not a floor task, CoFFiE reaches 61.49 against 55.80 for PERFT-R (+5.69, Table 2).
  2. Scope of the comparison. As stated in the introduction, this paper explores what else in an MoE can be a target of fine-tuning besides the weights themselves, and studies the coordination among the active experts as one such target, with the pretrained experts and the router kept frozen. Our claim is accordingly made against MoE-specific PEFT baselines, and we compare with PERFT. Full fine-tuning, ESFT and condenser experts train the experts themselves, and LoRA on all linear layers is a general method that does not depend on the MoE structure. They adapt the weights, which is a different question from the one we study. We find the reviewer's suggestions inspiring, and how our findings carry over to that setting will be our future work.

W3: Related work.

We sincerely thank the reviewer for these references. They are closely related to our work, and we discuss all of them in the revised paper (Appendix R).

  1. Condenser experts [3]. This is the closest prior work on consolidating information across sparsely activated experts. It is a supervised fine-tuning method that designates a few existing experts as condenser experts, always selected yet still gated per token, and learns routing biases that gradually move the experts less useful for the task out of the selected set. CoFFiE works in the parameter-efficient setting. It keeps all experts and the router frozen, and trains only a small shared operator that refines the outputs of the active experts using their consensus.
  2. SMT [5] and DoRA [1]. Both adapt individual weight matrices. SMT fine-tunes the sub-matrices with the most significant gradient updates, and DoRA decomposes each weight into magnitude and direction. In an MoE, a block inside an expert is updated only by the tokens routed to that expert, whereas our operator is shared by all active experts and is trained on all tokens.
  3. [2] and DenseMixer [4]. [2] compares LoRA with full fine-tuning on math and code; full fine-tuning is the setting discussed in Q3. DenseMixer improves MoE post-training through a more precise gradient for the router; CoFFiE leaves the router unchanged.

Q4: Decode latency and throughput relative to the frozen model and PERFT-R.

We thank the reviewer for raising this. We have measured decode latency and throughput for the frozen model, PERFT-R, PERFT-E and our method (table in General Response C). With the default n=3, decoding is within 6% of PERFT-R and faster than PERFT-E, and throughput at batch size 32 is 263 tok/s against 284 for PERFT-R.

We are grateful for the reviewer's detailed and constructive comments, which have improved the paper. We are happy to discuss any remaining point, and we hope these results are useful for the reviewer's final judgement.

We thank the reviewer again for the review. We would be grateful to know whether our response addresses the reviewer's concerns, and we are glad to clarify anything further.

Thank you for the detailed response. It addresses some of my concerns, and I have raised my score accordingly. I still consider the paper borderline. My remaining concerns are the novelty of the overall approach and what the fixed-point formulation contributes to the work.

We sincerely thank the reviewer for the reply and for raising the score. On the two remaining concerns:

Novelty. We ask what else in an MoE can be a target of fine-tuning besides the weights themselves. Our results show that the coordination among the experts is one such target. The learned negative α also points to redundancy across layers (General Response B), which may offer a further perspective for future MoE fine-tuning and the design of MoE models.

Fixed-point formulation. Iteration follows naturally from our consensus, each expert is refined using the consensus of the refined experts, so the refined outputs are defined by a fixed-point equation. This formulation improves accuracy, by +1.12 in the cross-layer version over three seeds (General Response A). It also needs much less training memory than unrolling, 26 GB against 37 to 66 GB (General Response C).

Reviewer_HVU2OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a new fine-tuning method that refines the representations of an MoE model using fixed-point iteration with a consensus-based mechanism. An interlayer mechanism is also introduced. The fixed-point iteration is theoretically investigated and motivated by consensus among the selected experts to refine their representations. Empirical results show that the proposed method improves over PERFT and LoRA baselines across multiple benchmarks, with varying degrees of improvement.

Critical strengths
  • The fixed-point iteration for updating expert outputs based on interactions among the selected experts is interesting.
  • The paper examines the convergence of the fixed-point iteration both theoretically and empirically.
  • The interlayer mechanism, although seemingly simple, appears to account for a substantial share of the improvement. The paper also acknowledges that it performs well even without fixed-point solver iterations.
Critical weaknesses
  • The improvement from in-layer fixed-point iteration over other baselines is small, as demonstrated in Table 9 in Appendix G. With comparable numbers of trainable parameters, it achieves 69.1869.18, compared with 68.7968.79 for PERFT-E and 68.8168.81 for PERFT-R, an improvements of 0.390.39 and 0.370.37 percentage points, respectively. These single-seed gains are modest. Using approximately the same number of parameters while requiring additional refinement computation with n=3n=3, the method introduces substantially more methodological complexity for a limited improvement, making its practical value unclear.
  • Much of the improvement appears to come from the simple interlayer mechanism rather than the elaborate Picard fixed-point formulation. Even so, the interlayer mechanism is explained only briefly, in seven lines (201–207) in Section 3.3.
  • The main novelty, as presented in the paper, clearly centers on the fixed-point consensus formulation. However, the stronger empirical gains appear to come from the simpler interlayer mechanism. Consequently, the motivation and theoretical analysis do not clearly explain the strongest benchmark improvements.
  • The iterative formulation adds inference computation that scales with the number of iterations, but neither inference latency nor inference memory usage is reported.
Questions for authors
  • Can the authors evaluate more clearly how much improvement comes specifically from fixed-point iteration compared with other baselines? If most of the improvement comes from the interlayer mechanism, how should the framing of the paper change?

  • The fixed-point iteration can be interpreted as adding repeated, weight-tied refinement layers on top of the existing expert outputs. What does the fixed-point framework contribute beyond this interpretation?

  • Since the refinement network is trained using gradient descent, what practical benefit comes from treating these repeated updates as an equilibrium solve rather than a finite stack of shared layers?

3 comments
AuthorsOpenReview ↗

Response to Reviewer HVU2

W1 / W3 / Q1: How much improvement comes from fixed-point iteration? If most comes from the interlayer mechanism, how should the framing change?

We thank the reviewer for these questions. The fixed point is not an extra module added on top of the method. Each expert is refined using the consensus of the refined experts, so the definition is circular, and such a circular definition is a fixed-point equation, which we solve by iteration (Section 3.2). Our new results show that iteration is effective with the cross-layer consensus as well, where it gives its largest gain (General Response A).

On the size of the gain and the complexity of the method (W1), the cross-layer version is our best configuration (General Response A). CoFFiE is also simple to use. It adds a single operator per layer whose only size parameter is its width, and we use the same structure on all three backbones. PERFT instead comes in several variants, and for each one has to choose how many adapters to add, how many to activate per token and their rank.

W2: The interlayer mechanism is explained in only seven lines in Section 3.3.

To explain the cross-layer mechanism further:

  1. Definition. Each MoE layer keeps a state m^ℓ = α·m^(ℓ−1) + (1−α)·c^ℓ, which combines the current consensus c^ℓ with the state of the previous layer through one learnable scalar α per layer. The operator reads m^ℓ in place of c^ℓ; everything else, including the iteration, is unchanged.
  2. What it does. After training α is negative, so the state emphasises how the current consensus differs from that of earlier layers (General Response B).
  3. Cost. The state is passed to the next layer without gradient, so training memory is unchanged (General Response C).

W4: Inference latency and memory are not reported.

We have measured inference latency and memory for all methods (General Response C). The default configuration is within 6% of PERFT-R in decoding time and faster than PERFT-E. Inference memory is essentially unchanged at batch size 1, and grows by at most 1.4 GB at batch size 32.

Q2 / Q3: Iteration can be viewed as stacked weight-tied refinement layers. What practical benefit comes from treating it as an equilibrium solve?

In the forward pass the two are the same computation. The benefit of the fixed-point view is in training, where we backpropagate only through the last iteration instead of every iteration, so training memory does not grow with the number of iterations (General Response C). Constant-memory training comes from deep equilibrium models [a], and the one-step gradient we use follows [b].

[a] Bai et al., Deep Equilibrium Models, NeurIPS 2019.

[b] Fung et al., JFB: Jacobian-Free Backpropagation for Implicit Networks, AAAI 2022.

We thank the reviewer for the time spent on our paper. We hope the three-seed results and the new cost measurements resolve the main concerns, and we would welcome any follow-up question. We would appreciate it if the reviewer could revisit the assessment in light of these results.

Reviewer_HVU2OpenReview ↗

Reply to author rebuttal

I thank the authors for clarifying runtime and memory usage. These concerns are resolved: runtime is comparable to the evaluated baselines, with lower training memory than explicitly unrolled refinement layers.

The added ablation shows gains from iteration beyond cross-layer propagation in one setting. However, systematic comparisons with and without iteration across all main settings are needed to establish this central contribution, as other reviewers have also noted.

These revisions would be difficult to complete and reassess within the rebuttal period. Although I recognize the paper’s merits, I unfortunately cannot recommend acceptance and decide to keep my score unchanged. I encourage the authors to revise the experiments and framing for a future submission.

We thank the reviewer for the reply and for recognising the merits of the paper, and we are glad that the runtime and memory concerns are resolved. We value the reviewer's suggestions on the experiments and the framing. If the reviewer has any further thoughts on which settings would be most informative for the comparison with and without iteration, we would be very grateful to hear them.

Reviewer_R16DOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoFFiE, a parameter-efficient fine-tuning method for pretrained Mixture-of-Experts (MoE) models. Instead of modifying individual experts or the router, CoFFiE introduces a lightweight shared refinement network that conditions each active expert's output on a route-weighted consensus of the selected experts. The refinement is formulated as a coupled fixed-point problem and trained using a one-step gradient approximation. The authors further introduce a cross-layer variant that propagates consensus information between MoE layers.Experiments on OLMoE-1B-7B, Mixtral-8×7B, and Qwen3-30B-A3B demonstrate improvements over the evaluated PEFT baselines on commonsense and arithmetic reasoning tasks. The paper also provides parameter-matched ablations, convergence analysis, and computational-cost measurements.

Critical strengths

The paper presents an interesting approach to MoE adaptation by focusing on coordination among frozen experts rather than independently updating their parameters. The method achieves encouraging results across three MoE architectures with relatively small trainable-parameter budgets. The parameter-matched ablations are carefully designed and provide useful evidence for the contributions of consensus information and cross-layer propagation. I appreciate the authors' detailed convergence analysis and their transparent discussion of the limitations of the proposed method.

Critical weaknesses

The contribution of fixed-point iteration is somewhat unclear in the cross-layer variant. According to Appendix I, the cross-layer method achieves comparable performance with n=0 and n=3 iterations. This raises the question of whether the equilibrium formulation is essential to the strongest configuration. The behavior of the cross-layer coefficient is not well understood. The method benefits primarily from negative initialization of the coefficient α. Since negative values turn the cross-layer update into a form of extrapolation rather than ordinary averaging, the underlying mechanism may differ from the consensus interpretation presented in the paper.

Questions for authors

Appendix I shows that the cross-layer variant achieves 69.01 average accuracy with n=0 and 68.78 with n=3. Does this suggest that cross-layer information propagation already provides most of the benefit of iterative consensus refinement? I would be interested in the authors' interpretation of this result. The negative initialization of α appears important for performance. Could this indicate that the model benefits from emphasizing differences between consecutive layers rather than simply aggregating their consensus information? An intuitive explanation of this behavior would be valuable.

1 comment
AuthorsOpenReview ↗

Response to Reviewer R16D

Q1: With the cross-layer state, n=0 and n=3 are comparable. Does cross-layer propagation already provide most of the benefit of iterative refinement?

We thank the reviewer for this insightful question. Our new results suggest that cross-layer propagation does not replace iteration. Iteration remains effective in the cross-layer version and brings a clear gain. Please see General Response A.

Q2: Does negative α indicate that the model benefits from emphasising differences between consecutive layers rather than aggregating? An intuitive explanation would be valuable.

We thank the reviewer; this reading matches what we observe. With a negative α, the operator also receives the difference between the current consensus and the earlier layers. We hypothesise that taking the difference reduces redundancy across layers. Prior work reports that in LLMs the hidden states entering and leaving many layers are highly similar [a], and that in pre-trained MoE models the experts selected in consecutive layers are strongly correlated [b]. The consensus of consecutive layers is therefore likely to overlap, so averaging it over layers smooths it out and repeats information that the residual stream already carries. Taking the difference removes this shared part and keeps what the current layer adds.

[a] Men et al., ShortGPT: Layers in Large Language Models are More Redundant Than You Expect, arXiv:2403.03853, 2024.

[b] Yao et al., Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference, IPDPS 2024.

We hope these answers address the reviewer's concerns, and we would be glad to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.