- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper is an empirical study of training-free test-time fusion of frozen specialist models. The main focus is whether successes that each specialist already had are preserved after combination. The authors do not propose new algorithm, and count retained / harmed / rescued cases instead of only reporting accuracy.
Critical strengths
-
Firstly. the motivation is practical. Aggregate accuracy cannot show which successes are kept or lost after combination, and retained / harm / rescue counting is a useful reporting practive for fusion research.
-
Second wordgame seperates evaluation axes cleanly. the deterministic simulator lets them check format, final state, and reported transitions separately. It shows clearly the case of right format wrong contents.
-
Finally, the confirmation panel is frozen separately. Fixing 96 inputs before seeing development results and reproducing the main contrast on them is good practice.
Critical weaknesses
-
The main wordgame results are almost trivial from the setup. Specialists have completely disjoint skills, so any fusion that does not put most weight on the home expert must fail. It is expected that equal-weight fusion and parameter merging are near 0%, and FixFuse equals HOME because it is basically HOME routing. This does not represent realistic case where experts abilities partially overlap.
-
The fixed vs dynamic contrast may be about fallback, not weighting. In D75, only 1 step out of about 98k tokens used normal pooling, and the rest was token-string-prefix fallback or main-EOS bypass. So FixFuse and D75 are basically main-expert-only decoding, which is why they equal HOME. Bit fallback rates for FixFist, DynFuse, and CoRE themselves are not reported. Whithout this I cannot accept the interpretation of main-weight protection vs auxiliary allocation, and the difference may just be when pooling happens.
-
Why fallback happens so often is not explained. It seems to come from interaction of top-10 support and token-string prefix rule, which depends heavily on implementation details. If the key result depends on an implementation artifact, it is hard to draw general conclusions.
-
DynFuse is not very representative. DynFuse is the authors’ ablated version of CoRE without token-level factors, while full CoRE works well (697). So DynFuse failure may be a problem of this specific simplification, not of “dynamic weighting” in general. There is no analysis of what exactly the CoRE token factors prevent.
-
The key contrast is not tested in the external setting. FixFuse and DynFuse are only run on WordGame, and external setting only evaluates EMFuse, CoRE, Mean, TIES, and PackLLM. External results also use matched subsets of 96 (AIME only 30), so most CIs overlap and no clear pattern appears. The only meaningful comparison is basically the 400-problem LCB.
-
Novelty and takeaways are weak. That right format is not correctness, that same accuracy can come from different solved problems, and that fused model can lose to the best expert are already known observations. The negative flip concept comes from prior work (Yan et al., 2021), and authors also say it is not a new metric. Because every claim is heavily hedged, there is almost no practical recommendation left.
Questions for authors
Basically, please refer to the critical weaknesses first.
-
How much do FixFuse outputs match HOME outputs at the token level? Is it basically the same as HOME decoding?
-
Under what exact condition does token-string-prefix fallback trigger? Why does fallback happen on almost every token?
-
The only difference between CoRE and DynFuse is the token-level factors. How exactly do these factors prevent DynFuse’s failure? Did you analyze the token position where the failure first appears?
-
Can you also run FixFuse and DynFuse in the external (Qwen3-4B) setting?
-
Do the results hold on a WordGame variant where specialists’ abilities partially overlap (for example, off-family accuracy is not zero)?
-
If specialists are retrained with multiple seeds and equal training budgets, is the fixed vs dynamic gap reproduced?