- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Mixture of Outputs (MoO), a decoding-time framework that combines a frozen receiver’s logits with those of frozen specialized donors. A lightweight gate uses the receiver’s hidden state to predict token-dependent donor strengths, with vocabulary alignment and logit-scale calibration supporting fusion.
Critical strengths
- Method is good, lightweight, compositional adaptation. MoO learns token-dependent donor weights while keeping the receiver and donors frozen. Independent gates allow multiple donors to contribute simultaneously, offering a practical way to reuse specialized models without updating the receiver.
- The visualization is done very well, making it relatively easy to understand the motivation and methods of the paper, and the result presentation is also relatively beautiful. If possible, could please introduce what tools were used to create these figures?
Critical weaknesses
- Insufficient reproducibility. The paper does not adequately specify the gate-training datasets, cross-vocabulary projection procedure, or safety-evaluation criteria, and further understanding of the specific implementation is hindered by a lack of supplementary materials.
- Limited evidence of robust superiority. Approximately 53% of the headline 14.27-point improvement comes from MBPP, where the receiver scores only 0.5%. This unusually low baseline requires investigation, and small reported gains need statistical validation. This is my main concern.
Questions for authors
See the Weaknesses.
2 comments
Thank you for the author's detailed response. My main concern has been resolved. Considering the overall quality of this work, I would like to maintain my rating.
Response to Reviewer BvgY
We thank the reviewer for finding MoO a lightweight, compositional way to reuse specialized models while keeping both receiver and donors frozen, and for the kind words on the figures and result presentation. We address the two weaknesses below.
W1: Reproducibility.
Thank you. We address the three missing items in turn: (1) gate-training data, (2) the cross-vocabulary mapping and (3) the safety-evaluation criteria.
(1) Gate-training data. Donor-generated, correctness-filtered examples: 1,736 for the single AceMath donor (1,000 math, 236 code, 500 general), 3,472 for the two-donor setting, and 860 for alignment. Reasoning questions come from training splits; AdvBench and XSTest items 1–200 are reserved for evaluation.
(2) Cross-vocabulary mapping. We correct the submission: the Gemma/Qwen map is a deterministic sparse token-index mapping built from the two vocabularies, not a learned projection. Donor probabilities are summed into the receiver vocabulary through this index.
(3) Safety evaluation. The first 200 AdvBench behaviors in the user chat template, with no jailbreak suffix; refusals are detected by a fixed list of refusal markers in the first 40 characters, and ASR is the share not refused. Over-refusal uses the first 200 non-contrast XSTest safe prompts with the same rule.
Full details are in our responses to Reviewer Q9v7 (Q2) and Reviewer YQeL (Q1, Q2).
Revisions: Appendix A.3–A.6 (L823–913) now specify the gate-training data, the cross-vocabulary mapping, the safety-evaluation protocol and the training and baseline settings, and Section 3.2 (L213–215) corrects the mapping description. We will release anonymized code for gate training, fusion decoding and evaluation.
W2: MBPP's share of the headline gain and statistical validation.
Thank you; we agree with this analysis, and the estimate is correct (68.0 / 9 = 7.56 of 14.27 points). We investigated it in four parts: (1) the cause of the 0.5% baseline, (2) whether MBPP is a reliable measure here, (3) the gain without MBPP and (4) statistical validation.
(1) Cause of the 0.5% baseline. Our MBPP protocol is zero-shot: the prompt asks for a single Python code block and shows the first test case; we extract the first fenced code block (or the whole completion if there is none), run all tests with an 8-second timeout, and allow 1,024 new tokens. In 198 of the 200 Qwen2.5-Math-7B outputs no Python function can be extracted. The cause is the zero-shot prompt rather than the extraction rule: with a three-shot prompt, the receiver scores 66.5% and MoO 65.5%. The zero-shot MBPP gain thus reflects instruction following transferred from the donor, not improved coding.
(2) MBPP is not a reliable measure for this receiver. Re-training the gate with two more seeds gives MBPP 0.010 and 0.165, versus 0.685 for the reported seed. We therefore no longer rely on MBPP and no longer report the 14.27-point figure.
(3) The gain without MBPP is robust across seeds. Excluding MBPP, MoO improves Qwen-Math from 0.4836 to 0.5682 ± 0.0090 over three seeds (+8.46 points). The gain holds for every seed (+7.55 to +9.36) and comes from the math benchmarks (GSM8K 0.575 → 0.860 ± 0.053; MATH500 0.530 → 0.710 ± 0.048).
(4) Statistical validation. Per-seed results are in our response to Reviewer YQeL (Q4). We are re-running evaluations with per-example logging to compute paired bootstrap intervals.
Revisions: The abstract (L24–26) and introduction (L103–104) report the seed-averaged eight-task gain. Appendix A.5 adds the MBPP protocol, the analysis of the receiver's outputs and the three-shot results (L871–891), and Section 4 adds the per-seed results (Table 4, L416–431). Paired bootstrap intervals will be added once the re-evaluation is complete.