acceptodds
Under review as a conference paper at ICLR 2027

Mergodic: Almost-Lossless Model Merging

Abstract

Fine-tuning a shared pretrained model yields a rapidly growing population of task-specific experts, and model merging has emerged as the standard way to consolidate them into a single deployable artifact without retraining. Existing approaches face a fundamental tension. Static merging methods collapse all experts into one set of weights and are compact, but parameter interference causes accuracy to degrade as the number of tasks grows. Task-conditioned methods, often implemented with routing mechanisms, recover much of this lost accuracy by retaining task-specific signal, but they reintroduce exactly what merging set out to remove: per-task masks, adapters, or expert modules whose storage scales with the number of tasks. We ask whether a single shared representation can instead regenerate any requested expert on demand. We answer in the affirmative with Mergodic, a task-conditioned merging method grounded in a classical result from ergodic theory: by the Kronecker-Weyl equidistribution theorem, the orbit of a one-dimensional irrational rotation is dense in the -dimensional torus, so the cross-task values at any parameter coordinate can be approximated by a single scalar phase. Mergodic discretizes this phase into one integer code per coordinate—shared across all experts and independent of the number of tasks—and reconstructs each expert through a deterministic, task-conditioned rotation map. The result is an almost-lossless merge: experts are recovered with negligible reconstruction error at a fraction of the storage of the original collection. We evaluate Mergodic under both expert-availability regimes: offline merging, where all experts are available simultaneously, and continual merging, where experts arrive sequentially and previous checkpoints are discarded. Across vision and natural language benchmarks, Mergodic attains state-of-the-art performance while remaining highly compact.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Model Merging, Ergodic Theory, Task-conditioned merging, Router-based merging

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_q5WmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper jointly encodes task vectors from multiple experts into a shared integer representation. At each parameter coordinate, a bounded integer indexes a deterministic trajectory that approximates the values across experts. A requested task index is then used to recover the corresponding update. This provides a compact, task-conditioned representation rather than one fixed set of merged weights for all inputs.

Critical strengths

The storage–performance trade-off is convincing in the evaluated settings. For 14-task ViT-L/14 merging, Table 1 reports 92.35 versus 92.62 for independently stored experts, while reducing storage from 14× to 1.5×. Importantly, Table 10 also compares compression methods at the same storage budget: Mergodic reaches 84.82, versus 83.34 for learned vector quantization and 83.42 for truncated SVD. This comparison makes the practical contribution clearer than comparisons with static merging alone.

Critical weaknesses

The coding construction has a close precedent in Hyper-Compression, which is cited in the appendix. The paper should make clearer which parts of the codec are inherited and what is new in its cross-expert organization. I view the contribution primarily as a useful representation and application of this construction, rather than a new coding principle. The near-lossless characterization also needs a more qualified scope. For full-finetuning GLUE, the average drops from 85.19 to 83.75, and MNLI drops from 87.20 to 79.17. These results do not negate the storage benefit, but they show that the small average losses in the vision experiments do not describe every evaluated setting.

Questions for authors

Which elements of the coding procedure are new relative to Hyper-Compression, beyond organizing coordinates across experts? For the MNLI result, what can the existing analysis establish about routing error versus parameter reconstruction error? A clear accounting would help delimit the near-lossless claim.

4 comments
AuthorsOpenReview ↗

Part (1/2): Novelty Relative to Hyper-Compression (Weakness 1 and Q1)

We sincerely thank the reviewer for their thoughtful and constructive feedback and for the opportunity to further clarify our contributions. We acknowledge that Hyper-Compression previously explored irrational winding for neural network compression, as already acknowledged and cited in our Related Work section. In response to the reviewer's suggestion, we have expanded this discussion in the revised manuscript to clarify the methodological distinctions and contributions of Mergodic. We will also move the Related Work discussion from the appendix into the main paper in the camera-ready version.

Beyond jointly encoding parameters across multiple experts, rather than compressing parameters within a single model as in Hyper-Compression, Mergodic introduces several methodological and theoretical contributions:

  1. Task-conditioned decoding and continual merging: We develop a reconstruction framework that recovers individual expert updates on demand and supports incorporating new experts through decode–append–re-encode operations, without retaining previous expert checkpoints.

  2. Finite-budget theoretical analysis: Beyond asymptotic density, we establish a universal lower bound and a conditional matching upper bound on the worst-case reconstruction error, explicitly characterizing how the required code budget scales with the number of experts and desired precision.

  3. Efficient GPU implementation: We develop a parallel nearest-code search that exploits the shared codebook structure to encode large task-vector matrices efficiently, together with lightweight deterministic reconstruction.

Thus, our contribution is not a new irrational-rotation principle, but a task-conditioned and continual merging methodology with explicit finite-budget analysis and an efficient implementation.

AuthorsOpenReview ↗

(Part 2/2): MNLI Performance Degradation (Weakness 2 and Q2)

We thank the reviewer for highlighting the MNLI performance degradation and requesting a breakdown of its causes.

To distinguish routing errors from parameter reconstruction errors, we evaluated Mergodic using both ground-truth task indices and the learned router, while keeping the same expert checkpoints.

ConfigurationMNLI Accuracy
Original MNLI expert87.20
Mergodic (ground-truth task index)81.09
Mergodic (learned router)79.17

Of the 8.03 percentage-point total degradation, 6.11 points arise from parameter reconstruction, while the remaining 1.92 points are attributable to routing.

We further investigated the reconstruction-related degradation by evaluating four different task orderings (frequency assignments). Across these settings, MNLI generally exhibits one of the highest reconstruction errors among the tasks, although its error is not substantially larger than that of the other experts. This suggests that MNLI's larger accuracy degradation may reflect a combination of relatively higher reconstruction error and greater sensitivity to parameter perturbations. The latter is a plausible explanation rather than a confirmed cause, since similar reconstruction errors do not necessarily translate into comparable downstream accuracy changes across tasks.

We will further clarify that the near-lossless characterization describes the strong performance retention observed across many evaluated settings, rather than guaranteeing negligible degradation for every individual task.

We thank the reviewer again for their constructive feedback and would be happy to clarify or address any remaining concerns.

Thank you for the detailed response and additional analysis. The relationship to Hyper-Compression and the paper’s contributions in task-conditioned decoding, continual merging, and finite-budget analysis are now clearer. The additional MNLI comparison also clarifies how much of the performance loss comes from reconstruction versus routing. My concerns have been largely addressed, and my overall assessment remains unchanged.

We sincerely thank the reviewer for their thoughtful feedback and for taking the time to consider our responses and additional analyses. We appreciate their positive assessment and constructive suggestions, which have helped us improve the clarity of the paper.

Reviewer_rYPoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Mergodic presents an unusual and useful approach to storing and recovering multiple experts through a shared integer representation. The paper clearly formulates its task-conditioned setting and supports it with extensive experiments and appendices. The main limitations concern the distinction from static capability merging, the practical relevance of the asymptotic theory, and incomplete evidence about deployment and continual reconstruction at larger task counts.

Critical strengths
  1. The formulation addresses a genuine storage problem. Sections 2-3 clearly distinguish a shared compressed representation from a single static model and describe a concrete, deterministic decoder.

  2. The mathematical motivation is coherent and its limits are acknowledged. Appendix A.8 explicitly derives the task-count/precision trade-off and distinguishes the conditional finite-budget upper bound from what is proved for the chosen frequencies.

  3. The evaluation goes beyond basic merging baselines. The paper includes task-conditioned competitors, matched-storage compression comparisons in Table 10, routing results, and measured reconstruction latency. Table 1 shows strong retention across 8, 14, and 20 tasks.

Critical weaknesses
  1. The strongest claims require a narrower interpretation of merging. Mergodic reconstructs one selected expert rather than producing a fixed model that simultaneously combines capabilities. Comparisons with TA and TIES therefore involve different inference assumptions. Twin-Merging, EMR, and Table 10 are more relevant comparisons, but the learned cross-expert quantization comparison is restricted to one vision setting. The claim should emphasize compressed expert recovery, and the most relevant matched-storage comparisons should be extended across the principal settings.

  2. Asymptotic density does not establish useful accuracy at the deployed code width. Appendix A.8 correctly acknowledges that maintaining worst-case precision requires code width growing with task count, and that the finite-budget upper bound is not established for the specific frequencies used. Consequently, the fixed-width success depends on empirical task-vector structure. Sensitivity to task ordering and frequency choice, and per-task reconstruction-error distributions at larger task counts, are insufficiently characterized. This is a gap in practical explanation, not an information-theoretic contradiction.

  3. Deployment and continual-retention evidence remains incomplete. Table 6 usefully reports routing and reconstruction latency, but storage ratios do not measure peak serving memory or throughput when consecutive requests select different experts. Table 5 uses a shared arrival order without showing order sensitivity or how repeated decode/re-encode errors accumulate for older tasks. Clarifying these costs and reporting task-wise retention would better support the practical and continual-merging claims.

Questions for authors
  1. Matched comparisons: Do Mergodic, Twin-Merging, EMR, and the compression baselines use the same routing data and predicted task indices? How are task heads, codebooks, normalization statistics, and routers counted in each storage budget?

  2. Finite-budget behavior: Were any alternative frequency assignments or task orders evaluated? Please clarify whether the choice between the squared-error and maximum-error objectives was made using a separate validation set.

  3. Serving and continual updates: What objects must coexist in GPU memory during expert reconstruction, and does Table 6 include their allocation and transfer costs? Are task-wise or arrival-order results already available for Table 5?

7 comments
AuthorsOpenReview ↗

(Part 1/4): Matched Comparisons and Storage Accounting (Weakness 1 and Q1)

We thank the reviewer for recognizing the value of our shared compressed representation and extensive evaluation. We appreciate the suggestions for further strengthening the comparisons.

1. Extending the matched-storage comparisons.

We agree that task-conditioned reconstruction is distinct from producing a single static model. Our objective is specifically to consolidate multiple experts into a compact shared representation while retaining the ability to recover their task-specific capabilities. Accordingly, Twin-Merging, EMR, and the compression baselines provide particularly relevant comparisons.

To further strengthen the empirical evaluation, we have extended the learned cross-expert VQ baseline to all principal vision settings (8, 14, and 20 tasks), as well as both LoRA and fully fine-tuned NLP settings, in the revised manuscript. Mergodic continues to achieve the best performance under the evaluated storage constraints. We will also extend the remaining individual-compression baselines to the main vision, NLP, and generative settings for completeness.

We would also like to highlight that joint cross-expert VQ is itself an additional model-merging baseline introduced in our work. Rather than quantizing experts independently, it represents their values jointly at each parameter coordinate using a shared codebook. To our knowledge, this formulation has not previously been evaluated as a task-conditioned model-merging approach. We include it as an informative ablation of the proposed shared-representation framework.

2. Q1: Routing consistency and storage accounting.

Do the merging baselines use the same routing data and predicted task indices?

Yes. All task-conditioned merging baselines use the same learned router, calibration data, and predicted task indices. The individual-compression methods in our ablation use the ground-truth task index (oracle routing) to isolate compression quality from routing errors, allowing us to evaluate their intrinsic capability-preservation performance independently of the router.

How are task heads, codebooks, normalization statistics, and routers counted?

These components, including task counts, normalization statistics, codebooks, and two-layer MLP routers, together require approximately 2–10 MB across our evaluated settings, representing negligible overhead relative to model checkpoints. We will clarify this accounting more explicitly in the revised manuscript.

AuthorsOpenReview ↗

(Part 2/4): Finite-Budget Behavior and Frequency Robustness (Weakness 2 and Q2)

We thank the reviewer for the constructive suggestions regarding practical reconstruction behavior.

1. Finite-budget reconstruction fidelity.

We agree that asymptotic density alone does not guarantee negligible reconstruction error at a fixed code width. Our theoretical analysis explicitly distinguishes the asymptotic result from finite-budget approximation, including the universal lower bound and the conditional upper bound.

The practical effectiveness of Mergodic at 16 bits is established empirically, rather than inferred solely from asymptotic density. Real task vectors exhibit structure and correlations that can make them more amenable to compression than arbitrary points in the unit hypercube. To characterize this behavior more thoroughly, we have added reconstruction-error measurements and distributions for 8, 14, and 20 tasks in the revised manuscript in Appendix A.13.

Furthermore, the storage–precision trade-off is explicitly controllable: increasing the integer bit budget improves the available approximation precision as the number of tasks grows.

2. Q2: Alternative frequencies and encoding objectives.

Were any alternative frequency assignments evaluated?

Yes. In addition to our default frequencies at=1/(π+t)a_t = 1/(\pi+t), we experimented with at=1/pta_t = 1/\sqrt{p_t}, where ptp_t is the tt-th prime number. We observed similar performance.

Was the choice between squared-error and maximum-error encoding made using a separate validation set?

No. We report both squared-ℓ2\ell_2 and ℓ∞\ell_\infty encoding objectives in Appendix A.11.3, making their respective performance directly available for comparison. Our experiments show that squared-ℓ2\ell_2 performs well for smaller task collections, while ℓ∞\ell_\infty becomes increasingly beneficial as the number of tasks grows.

When available, a small calibration set, such as the one used for router training, could be used to select the encoding objective based on downstream performance. For strictly data-free merging, both objectives can instead be evaluated using reconstruction errors computed directly from the task vectors, without requiring validation data. Although reconstruction errors do not guarantee superior downstream performance, our empirical results suggest that ℓ∞\ell_\infty becomes increasingly beneficial as the number of jointly encoded experts grows, providing a practical heuristic for objective selection.

Finally, the sensitivity to expert arrival order is addressed through additional continual-merging experiments in Part 4, where we evaluate multiple orders and provide task-wise retention measurements.

AuthorsOpenReview ↗

(Part 3/4): Serving Memory and Inference Efficiency (Weakness 3 and Q3)

We appreciate the reviewer's suggestion to further clarify GPU serving memory and expert-switching overhead. We conducted additional measurements on Llama-3.2-3B (BF16, NVIDIA H100) to complement the inference results already reported in Table 6.

1. Q3: What must coexist in GPU memory during reconstruction?

In our measured GPU serving configuration, the pretrained backbone and 16-bit integer codes are kept in GPU memory, while an additional GPU buffer is used to reconstruct and execute the selected expert during inference. Each occupies approximately 6.43 GB, while the codebook, normalization statistics, and router MLP contribute negligible additional memory.

ObjectGPU memory (GB)
Pretrained backbone (also used for routing)6.43
GPU buffer for reconstructing the selected expert6.43
16-bit integer codes6.43
Codebook, normalization statistics, router MLP<0.01

We additionally evaluated keeping the compressed codes in pinned CPU memory:

ConfigurationResident GPU memory (GB)Peak GPU memory (GB)Reconstruction per switch
Pretrained baseline6.436.56—
Mergodic (GPU-resident codes)19.3220.26100.13 ms
Mergodic (CPU-resident codes)12.9013.90216.86 ms

Thus, offloading the codes to CPU reduces peak GPU memory from 20.26 GB to 13.90 GB, at the cost of additional reconstruction latency. The difference between resident and peak memory is primarily attributable to temporary reconstruction buffers.

The pretrained backbone could also be offloaded to CPU, potentially reducing the resident GPU model weights to approximately 6.43 GB for the selected expert, excluding temporary buffers and inference memory. This configuration would introduce additional transfer overhead and was not included in our measurements.

These results clarify that GPU serving memory is deployment-dependent and should be distinguished from Mergodic's 2.0× persistent storage requirement for Llama-3.2-3B.

2. Q3: Allocation, transfer costs, and consecutive expert switching.

The GPU buffer used for the selected expert is allocated once and reused across task switches, avoiding repeated full-model allocations. With GPU-resident codes, reconstruction requires no host-to-device code transfers. Keeping the codes in CPU memory instead reduces GPU memory consumption by approximately 6.4 GB, with reconstruction increasing from 100.13 ms to 216.86 ms, including transfer costs.

Table 6 already reports routing, reconstruction, and end-to-end serving latency. To further address the reviewer's question regarding consecutive requests selecting different experts, we additionally measured end-to-end latency when routing and reconstruction are performed on every request. The resulting latency is 1370.0 ms versus 1233.0 ms for the pretrained baseline (1.11×). Consecutive requests selecting the same expert can reuse the reconstructed weights and avoid repeated decoding.

We have incorporated these additional measurements and clarified the memory–latency trade-offs in Appendix A.7 of the revised manuscript.

Reviewer_EJ97OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Mergodic, a task-conditioned merging method for experts fine-tuned from one pretrained model. It normalizes each task vector per matrix and per task, then treats the values of all experts at one parameter coordinate as a point in the unit hypercube. Motivated by the Kronecker-Weyl theorem, under which an irrational-rotation orbit is dense in the cube, it searches a finite prefix of the orbit for the nearest point and stores that point's integer index. With 16-bit codes, storage is 1.5 times one 32-bit model regardless of the number of tasks.

Critical strengths
  1. Joint coding across tasks is a simple, deterministic and cheap alternative to per-task masks or low-rank modules. Storage is set by one explicit parameter, decoding needs no search (74.5 ms on Llama-3.2-3B), and encoding is fast on one GPU. If the results hold under controlled comparison, this is a practical option for serving many experts.
  2. The evaluation is broad (vision, full and LoRA NLP, generative, continual), and the paper is candid about limits: it proves the universal covering-radius lower bound, states that its frequencies are not shown to satisfy the condition of the upper bound, and acknowledges the exponential worst-case budget in nn.
Critical weaknesses
  1. This is essentially a fixed, data-independent vector quantizer, and "almost-lossless" is not measured where it matters. A 16-bit code gives 16/n16/n bits per task per coordinate, only 0.8 at n=20n=20. The paper's own lower bound gives worst-case normalized error of at least 0.125 at n=8n=8 and 0.287 at n=20n=20, while its empirical check stops at n=6n=6. In a quick synthetic check (i.i.d. uniform targets, n=20n=20), a random codebook gave the same RMSE as the orbit (0.179 vs. 0.180), so it is unclear what the Kronecker-Weyl construction adds at this budget. Real coordinates are correlated, so this needs to be measured on actual task vectors.

  2. The comparisons do not clearly support the claimed advantage. With ℓ2\ell_2, the objective used at 8 and 14 tasks, Mergodic falls below Twin-Merging at 20 tasks (77.48 and 84.15 vs. 80.55 and 84.47); it leads only after switching to ℓ∞\ell_\infty. Learned VQ should have lower distortion than a fixed codebook of the same size, yet it scores below Mergodic (83.34 vs. 84.82 in Table 10), so either the baseline is weak or accuracy does not track reconstruction. EMR is charged a 32-bit shared vector, so part of its storage gap is just precision, and the continual baselines are single static models.

Questions for authors
  1. Can you report reconstruction error at b=16b=16 for n=8,14,20n=8,14,20, relative to the coordinate std? Does "negligible" hold there?
  2. How was the learned VQ in Table 10 built, and what was its distortion compared with the orbit? Why would a codebook fitted to the data do worse?
  3. Please add EMR with a 16-bit shared vector ((1.5+n/32)×(1.5+n/32)\times storage) as a matched-budget baseline.
6 comments
AuthorsOpenReview ↗

(Part 1/3): Theoretical Bounds and Reconstruction Fidelity (Weakness 1 and Q1)

We thank the reviewer for raising these important points. We have evaluated finite-budget reconstruction on task vectors from 8, 14, and 20 experts, complementing our theoretical analysis. We first clarify the theoretical result, then present the new empirical evidence.

1. Clarifying the theoretical lower bound. The reviewer correctly observes that Proposition 2 gives lower bounds of 0.125 and 0.287 for 8 and 20 tasks at 16 bits. However, an important distinction is that these are universal lower bounds on the worst-case covering radius over the entire unit hypercube, not lower bounds on the reconstruction error of individual task-vector coordinates or their average distortion. Specifically, for any codebook containing K+1K+1 points in [0,1]n[0,1]^n,

r∞(K)≥12(K+1)−1/n.r_{\infty}(K) \geq \frac{1}{2}(K+1)^{-1/n}.

Importantly, this bound applies to every finite-codebook method, including random and learned VQ, regardless of how the codewords are constructed. It establishes the fundamental limitation of uniformly approximating every possible point in a high-dimensional cube under a fixed bit budget. It does not imply that actual model parameters must incur these worst-case errors.

Mergodic does not claim to circumvent this limitation. Rather, Kronecker–Weyl equidistribution establishes the asymptotic density of our construction, while Appendix A.8 characterizes the fundamental finite-budget storage–precision trade-off.

2. Additional reconstruction experiments on real task vectors.

To directly address Q1, we evaluated Mergodic with 16-bit codes (65,536 codewords) on the actual ViT-B/16 task vectors for 8, 14, and 20 tasks. We report RMSE in both the normalized space xx in [0,1] and the original task-vector space τ\tau, together with NRMSE relative to the original task-vector standard deviation.

TasksRMSE(xx)RMSE(τ\tau)NRMSE
80.073650.0002450.399
140.126310.0004600.669
200.161480.0005750.825

These measurements show that absolute task-vector reconstruction errors remain small, while relative errors increase as more tasks share the same 16-bit code, consistent with our theoretical analysis.

NRMSE is relatively high partly because of min–max normalization: each tensor is normalized over its full range, which can be determined by a few extreme entries and span several standard deviations, while most entries lie near the center. The reconstruction error scales with this range rather than with the standard deviation σ\sigma, so it can be large relative to σ\sigma even when it is small relative to the full range.

Importantly, this trade-off is controllable: increasing the integer bit budget can improve reconstruction fidelity as the number of tasks grows, without introducing separate task-specific parameter modules. We also clarify that our use of "negligible" primarily refers to the small downstream performance degradation observed in most evaluated settings, rather than uniformly negligible parameter reconstruction errors.

3. Additional comparison with random VQ. We also extended the reviewer's synthetic comparison to actual model parameters. We evaluated a random VQ baseline using the same 65,536-codeword budget, normalization, and nearest-code search as Mergodic, including an origin codeword for a fair comparison.

Interestingly, Mergodic achieves 4.7%, 6.1%, and 3.7% lower original task-vector RMSE than random VQ for 8, 14, and 20 tasks, respectively. Moreover, these improvements are not confined to a few dominant parameter matrices: Mergodic achieves lower RMSE in 70.2%, 84.8%, and 81.5% of the layer-wise comparisons, respectively. This suggests that the advantage of our construction extends across much of the model rather than being driven by a small number of large layers.

These findings show a consistent aggregate advantage over the tested random codebook on task vectors. We agree that random codebooks can achieve comparable distortion on synthetic uniform targets; our contribution does not require the orbit to outperform every alternative codebook in finite-budget distortion.

4. Reconstruction distortion versus downstream performance.

Reconstruction distortion should be considered alongside the ultimate objective of model merging: preserving the capabilities of the original experts under severe storage constraints.

As demonstrated in Table 1, Mergodic retains performance close to that of the original experts across increasingly large task collections while requiring only 1.5× the storage of a single model, even when representing 20 experts. It also outperforms the evaluated merging baselines under the reported storage constraints. These findings highlight that strong downstream performance does not necessarily require negligible parameter-wise reconstruction error.

AuthorsOpenReview ↗

(Part 2/3): Learned VQ and Structured Codebooks (Q2)

We thank the reviewer for this question. Building on our reconstruction analysis in Part 1, we clarify the learned-VQ implementation and the advantages of our construction.

1. Learned-VQ implementation.

Our learned-VQ baseline uses one global codebook shared across all layers, trained using FAISS k-means with approximately 1.05 million sampled cross-task coordinate vectors and 20 iterations. Each coordinate is encoded by its nearest centroid, following the same normalization and reconstruction procedure as Mergodic.

Notably, this joint cross-expert VQ formulation is also an additional model-merging baseline introduced in our work. Unlike methods that quantize individual task vectors separately, it jointly encodes the values of multiple experts at each parameter coordinate, enabling task-conditioned reconstruction from a shared codebook. To our knowledge, this specific formulation has not previously been evaluated for model merging.

Importantly, while learning separate codebooks for individual layers could improve reconstruction, each additional codebook would increase storage, weakening the comparison at a fixed budget. Even with a single global codebook, storing K centroids of dimension n requires O(Kn) additional values, growing linearly with the number of tasks. In contrast, Mergodic generates its codewords analytically without persistent centroid storage, preserving its compact representation as the number of tasks grows.

2. Reconstruction error and downstream performance.

Our additional experiments reveal an interesting distinction between average and maximum reconstruction errors. Although learned VQ achieves lower RMSE in 60.9%, 54.3%, and 56.2% of layers for 8, 14, and 20 tasks, respectively, Mergodic achieves lower maximum absolute reconstruction error in 96.7%, 76.8%, and 62.3% of layers, respectively.

This is particularly interesting because k-means minimizes average squared reconstruction distortion, rather than the largest individual reconstruction errors. Furthermore, training a single global codebook on sampled coordinates may not adequately represent all layer distributions, especially relatively rare but potentially important parameter regions.

Since neural networks can exhibit heterogeneous sensitivity to parameter perturbations, larger reconstruction errors in functionally important directions may disproportionately affect downstream performance. This provides a plausible explanation for the observed difference: Table 10 shows that Mergodic outperforms learned VQ by 1.48 percentage points in average task performance, despite learned VQ's data-dependent optimization. The updated main tables in the revised pdf further demonstrate that this performance advantage extends across the evaluated vision and NLP settings.

3. Benefits of the Mergodic

Beyond its storage advantages, Mergodic requires neither training data nor codebook optimization . Its analytic construction enables efficient, highly parallel GPU encoding, deterministic task-conditioned decoding, and continual expansion without codebook retraining, while our theoretical framework characterizes its approximation properties and fundamental storage–precision trade-off.

We do not claim that the orbit is universally optimal in reconstruction distortion. Rather, combining more accurate, potentially data-adaptive quantization with Mergodic's compact representation is a promising future direction, potentially improving reconstruction while preserving storage efficiency.

Overall, Mergodic offers a distinct combination of strong downstream performance, favorable maximum-error behavior, compact storage, analytic decoding, and continual extensibility, without requiring learned codebooks.

AuthorsOpenReview ↗

(Part 3/3): EMR Storage Accounting and Continual Merging (Weakness 2 and Q3)

We thank the reviewer for these valuable suggestions.

1. EMR with a 16-bit unified task vector.

Following the reviewer's suggestion, we additionally evaluated EMR with its unified task vector stored in FP16 rather than FP32, reducing its storage cost to (1.5+n/32). The results demonstrate that using a 16-bit unified task vector preserves virtually identical performance:

ModelTasksEMR (FP32)EMR (FP16)
ViT-L/14890.1290.12
ViT-L/141490.1190.08
ViT-L/142082.9082.85
ViT-B/16883.9583.95
ViT-B/161485.0685.05
ViT-B/162078.8778.84

We also evaluated this modification on LoRA and fully fine-tuned NLP experts, observing unchanged average performance in both settings. We have incorporated these results and the corresponding storage accounting in the revised manuscript.

2. Continual merging baselines.

We appreciate the reviewer's observation that the existing continual baselines produce a single static model, whereas Mergodic supports task-conditioned reconstruction. To provide a more comprehensive comparison, we will additionally include continual versions of Twin and EMR, alongside our existing continual baselines.

3. Consistency under a fixed encoding objective.

Using ℓ∞\ell_\infty uniformly across all task counts (Appendix A.11.3: 86.31/91.14, 87.88/92.32, and 81.10/85.11 for 8, 14, and 20 tasks, respectively), Mergodic outperforms Twin-Merging, EMR, and learned VQ in every setting of Table 1. Thus, a single fixed encoding objective suffices across all evaluated settings, without requiring per-setting objective selection.

We hope these additional results and clarifications address the reviewer's concerns. We would be happy to address any remaining questions or suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.