- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CoFFiE, a parameter-efficient fine-tuning (PEFT) method for Mixture-of-Experts (MoE) models. While existing MoE PEFT methods typically adapt selected experts independently, CoFFiE trains a small shared network to jointly refine each expert's output by conditioning it on information from other co-activated experts. To ensure consistency between the refined expert outputs and their consensus, the authors formulate the refinement process as a coupled fixed-point problem. Experiments on three MoE models demonstrate improved performance with a small number of trainable parameters.
Critical strengths
1. Parameter-efficient and practical design.
CoFFiE employs a lightweight refinement network shared across active experts, allowing cross-expert information to be incorporated without increasing the number of trainable parameters proportionally to the total number of experts. Additionally, the proposed training procedure avoids backpropagating through the forward solver iterations, keeping the backward memory cost independent of the number of iterations. These properties make the approach practically appealing for fine-tuning large MoE models.
2. Evaluation across multiple MoE architectures.
The paper evaluates CoFFiE on three MoE backbones, demonstrating consistent improvements over the evaluated baselines on commonsense and arithmetic reasoning benchmarks. The paper also provides implementation details, code repository references, and ablation studies, which are valuable for reproducibility and understanding the proposed method.
Critical weaknesses
1. Further justification for consensus-conditioned and fixed-point refinement.
The paper motivates expert coordination by the lack of explicit interaction among co-activated experts, but provides limited analysis of why independent adaptation is problematic in practice, such as conflicting corrections or redundant updates.
Tables 3 and 7 support the usefulness of consensus-conditioned feedback, but do not clearly establish the additional benefit of fixed-point consistency. Accuracy is similar for n=1, 3, and 5, and the cross-layer variant performs well even at n=0. While the residual and Jacobian analyses support the solver’s numerical behavior, it remains unclear whether closer approximation to an equilibrium improves downstream performance. A comparison with finite-step refinement trained through explicit unrolling, under matched parameter and computational budgets, would help clarify this contribution.
2. Clarification of baseline selection and statistical reliability.
The experimental results across three MoE backbones are encouraging. The paper states that the baselines are re-evaluated under the same training and evaluation settings using the hyperparameters from Liu et al. (2024). However, it would be helpful to provide more details on how the specific baseline configurations were selected for each backbone. In particular, the evaluated PERFT variants and their configurations differ considerably across models, making it difficult to understand the selection criteria. Clarifying whether these configurations were selected based on prior results, parameter budget constraints, or additional validation would help establish the fairness of the comparisons.
Furthermore, Table 1 reports single-seed results. Although additional multi-seed results are provided for CoFFiE on OLMoE, corresponding statistical comparisons with the baselines are limited. Providing multi-seed results for the strongest baselines would strengthen the reliability of the reported improvements.
Questions for authors
1. Necessity of the fixed-point formulation.
The ablation results suggest that introducing feedback can improve performance, but they do not clearly demonstrate that multiple iterations or convergence toward a fixed point are necessary. In particular, n=1, 3, and 5 achieve statistically indistinguishable results, and the cross-layer variant already performs well with n=0. Could the authors clarify what benefit is uniquely attributable to the fixed-point formulation, beyond single-step consensus-conditioned refinement? Comparisons with simpler iterative refinement methods under matched computational budgets would be particularly helpful.
2. Baseline selection and statistical reliability.
Could the authors clarify how baseline configurations were selected for each backbone and whether comparable hyperparameter search budgets were used? In addition, could the authors provide multi-seed results for the strongest baselines, particularly for the comparisons in Table 1, to establish whether the reported improvements are robust?
Minor Comments
- Notation clarity (Section 3.2): The notation (E^\star) is introduced implicitly through the fixed-point equations. Explicitly defining before Eq. (3) would improve readability and make the formulation easier to follow.
- Inconsistent code repository URLs: The repository URLs in the Abstract and the Reproducibility Statement are inconsistent. Furthermore, the URL in the Reproducibility Statement appears to be inaccessible. Please verify the links and provide a consistent, accessible repository URL.
3 comments
We thank the reviewer again for the review. We would be grateful to know whether our response addresses the reviewer's concerns, and we are glad to clarify anything further.
While answering a follow-up question from Reviewer R16D, we obtained a result that also bears on the reviewer's Q2 about hyperparameter search, and we would like to share it here. PERFT-R does not benefit from a larger learning rate (68.86 at its original 1e-5, against 37.13 at 5e-4), whereas Table 1 reports CoFFiE at 2e-4, not at its best learning rate of 5e-4. We would be grateful to know whether our response addresses the reviewer's concerns, and we are glad to clarify anything further.
Response to Reviewer M5mx
W1 / Q1: (a) Why independent adaptation is problematic; (b) what benefit is unique to the fixed-point formulation beyond single-step consensus refinement; comparison with finite-step refinement trained by explicit unrolling at matched parameters and compute.
We thank the reviewer for these questions.
W2 / Q2: Baseline selection criteria and multi-seed results for the strongest baselines.
We thank the reviewer for these questions.
Minor comments. E* denotes the coupled fixed point defined by Eq. (3), that is, the refined expert outputs. The anonymous repository link in the abstract is the correct one, and we have confirmed that it is accessible.
We appreciate the reviewer's suggestions, in particular the comparison with explicit unrolling, which made the trade-off in our training procedure clearer. We remain available for further discussion, and we hope the reviewer will take the additional evidence into account in the evaluation.