- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper jointly encodes task vectors from multiple experts into a shared integer representation. At each parameter coordinate, a bounded integer indexes a deterministic trajectory that approximates the values across experts. A requested task index is then used to recover the corresponding update. This provides a compact, task-conditioned representation rather than one fixed set of merged weights for all inputs.
Critical strengths
The storage–performance trade-off is convincing in the evaluated settings. For 14-task ViT-L/14 merging, Table 1 reports 92.35 versus 92.62 for independently stored experts, while reducing storage from 14× to 1.5×. Importantly, Table 10 also compares compression methods at the same storage budget: Mergodic reaches 84.82, versus 83.34 for learned vector quantization and 83.42 for truncated SVD. This comparison makes the practical contribution clearer than comparisons with static merging alone.
Critical weaknesses
The coding construction has a close precedent in Hyper-Compression, which is cited in the appendix. The paper should make clearer which parts of the codec are inherited and what is new in its cross-expert organization. I view the contribution primarily as a useful representation and application of this construction, rather than a new coding principle. The near-lossless characterization also needs a more qualified scope. For full-finetuning GLUE, the average drops from 85.19 to 83.75, and MNLI drops from 87.20 to 79.17. These results do not negate the storage benefit, but they show that the small average losses in the vision experiments do not describe every evaluated setting.
Questions for authors
Which elements of the coding procedure are new relative to Hyper-Compression, beyond organizing coordinates across experts? For the MNLI result, what can the existing analysis establish about routing error versus parameter reconstruction error? A clear accounting would help delimit the near-lossless claim.
4 comments
(Part 2/2): MNLI Performance Degradation (Weakness 2 and Q2)
We thank the reviewer for highlighting the MNLI performance degradation and requesting a breakdown of its causes.
To distinguish routing errors from parameter reconstruction errors, we evaluated Mergodic using both ground-truth task indices and the learned router, while keeping the same expert checkpoints.
| Configuration | MNLI Accuracy |
|---|---|
| Original MNLI expert | 87.20 |
| Mergodic (ground-truth task index) | 81.09 |
| Mergodic (learned router) | 79.17 |
Of the 8.03 percentage-point total degradation, 6.11 points arise from parameter reconstruction, while the remaining 1.92 points are attributable to routing.
We further investigated the reconstruction-related degradation by evaluating four different task orderings (frequency assignments). Across these settings, MNLI generally exhibits one of the highest reconstruction errors among the tasks, although its error is not substantially larger than that of the other experts. This suggests that MNLI's larger accuracy degradation may reflect a combination of relatively higher reconstruction error and greater sensitivity to parameter perturbations. The latter is a plausible explanation rather than a confirmed cause, since similar reconstruction errors do not necessarily translate into comparable downstream accuracy changes across tasks.
We will further clarify that the near-lossless characterization describes the strong performance retention observed across many evaluated settings, rather than guaranteeing negligible degradation for every individual task.
We thank the reviewer again for their constructive feedback and would be happy to clarify or address any remaining concerns.
Thank you for the detailed response and additional analysis. The relationship to Hyper-Compression and the paper’s contributions in task-conditioned decoding, continual merging, and finite-budget analysis are now clearer. The additional MNLI comparison also clarifies how much of the performance loss comes from reconstruction versus routing. My concerns have been largely addressed, and my overall assessment remains unchanged.
Response
We sincerely thank the reviewer for their thoughtful feedback and for taking the time to consider our responses and additional analyses. We appreciate their positive assessment and constructive suggestions, which have helped us improve the clarity of the paper.
Part (1/2): Novelty Relative to Hyper-Compression (Weakness 1 and Q1)
We sincerely thank the reviewer for their thoughtful and constructive feedback and for the opportunity to further clarify our contributions. We acknowledge that Hyper-Compression previously explored irrational winding for neural network compression, as already acknowledged and cited in our Related Work section. In response to the reviewer's suggestion, we have expanded this discussion in the revised manuscript to clarify the methodological distinctions and contributions of Mergodic. We will also move the Related Work discussion from the appendix into the main paper in the camera-ready version.
Beyond jointly encoding parameters across multiple experts, rather than compressing parameters within a single model as in Hyper-Compression, Mergodic introduces several methodological and theoretical contributions:
Task-conditioned decoding and continual merging: We develop a reconstruction framework that recovers individual expert updates on demand and supports incorporating new experts through decode–append–re-encode operations, without retaining previous expert checkpoints.
Finite-budget theoretical analysis: Beyond asymptotic density, we establish a universal lower bound and a conditional matching upper bound on the worst-case reconstruction error, explicitly characterizing how the required code budget scales with the number of experts and desired precision.
Efficient GPU implementation: We develop a parallel nearest-code search that exploits the shared codebook structure to encode large task-vector matrices efficiently, together with lightweight deterministic reconstruction.
Thus, our contribution is not a new irrational-rotation principle, but a task-conditioned and continual merging methodology with explicit finite-budget analysis and an efficient implementation.