- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper addresses the mismatch between pretrained routing and downstream learning needs when expanding MoE models. ReconExp uses downstream gradients to select initialization donors and off-route distillation to broaden supervision for newly added experts. With pretrained parameters frozen and the active-expert count unchanged, the method updates approximately 1.45% as many parameters as full fine-tuning and outperforms the structurally matched expansion baseline on BFCL and WebShop.
Critical strengths
- Clear gains over a structurally matched baseline. ReconExp improves BFCL accuracy from 43.99% to 49.25% over LEGO and also improves WebShop performance, supporting its practical value for downstream adaptation.
- Convincing component-interaction analysis. Ablations show that SED helps only when paired with LGI, supporting the design rationale of establishing task-relevant initialization before expanding supervision coverage.
Critical weaknesses
-
The mechanistic evidence is not sufficiently independent. The gradient-coverage analysis compares task gradients with distillation gradients, rather than task-gradient coverage before and after adding SED. It therefore does not directly establish broader access to useful downstream learning signals.
-
Limited evaluation scope and baseline coverage. Experiments primarily cover one backbone and two task settings, while Expert-LoRA uses substantially fewer trainable parameters. This limits conclusions about generality and superiority over strong parameter-efficient adaptation methods.
Questions for authors
See weakness.
Thanks for the detailed response. I believe the author team has addressed my concerns. Therefore, I will maintain my positive score.