- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes FreeMOCA for class-incremental malware family classification without retaining past samples. The method adapts the previous consolidated model to a new task, then interpolates the parameters shared by the previous and adapted models. It copies the adapted parameters for new classifier outputs and uses the consolidated model to initialize the next task. The authors examine interpolation geometry, expand the recursive update, and derive a local loss bound to explain how the trajectory carries historical information. Experiments on Windows and Android malware benchmarks report competitive accuracy and retention with low training time and persistent storage requirements.
Critical strengths
- Practical problem and simple method. The replay-free setting addresses limits on retaining historical data when updating malware classifiers. FreeMOCA uses direct post-task interpolation without neuron alignment or variance repair.
- Performance and efficiency. In the main experiments, FreeMOCA outperforms the evaluated replay-free methods on both malware benchmarks and is competitive with replay-based approaches. Its training time and average persistent storage use are lower than those of the three replay-based methods in Table 4.
Critical weaknesses
-
Research contribution. FreeMOCA's update to shared parameters and the exponentially weighted expansion in Proposition 2 have the same form as those in Lookahead [1]. CLeWI also uses recursive post-task interpolation [2]. The paper does not clearly explain what new insight comes from removing replay, alignment, and repair and evaluating the simplified method on malware benchmarks.
-
Explanation of retention. The Consolidated segment in Figure 2 covers only the first λ fraction of the adaptation path. A lower barrier does not by itself mean lower loss on earlier tasks. The bound reduction in Eq. (14) assumes a fixed adapted endpoint, and the classifier-expansion term is not quantified. Figures 2–3 do not directly test how these analyses relate to the reported retention gains.
-
Task ordering and applicability. Compared with randomized ordering, dissimilar ordering in Table 7 lowers average accuracy from 65.2% to 22.86% on EMBER-Class but raises it from 63.7% to 67.86% on AZ-Class. Appendix B.4 attributes this contrast to feature reuse, parameter displacement, and interpolation barriers. It reports no corresponding measurements or baseline results under the same ordering, leaving the cause of the drop and the conditions for reliable consolidation unclear.
-
Experimental reporting. Table 2 reports 65.2±0.05% with default SGD, whereas Table 9 reports 69.09% for SGD and 64.19%–69.21% across initializations. The paper does not clarify whether the training settings and reporting protocols are comparable. The results for randomized task orders in Table 7 also exactly match Table 2's means and standard deviations, but it is unclear whether they come from the same runs.
[1] Lookahead Optimizer: k steps forward, 1 step back. NeurIPS 2019.
[2] Continual Learning with Weight Interpolation. CVPR 2024.
Questions for authors
-
CLeWI reports that interpolation has limited effectiveness without replay during training [2]. Which conditions make FreeMOCA effective on these malware tasks, and what new evidence supports that explanation?
-
Does interpolation improve earlier-task retention, and how does it affect new-task learning? Please compare earlier-task losses for both endpoints and the consolidated model in Figure 2, and report earlier-task and new-task accuracies separately in Figure 3.
-
With 50 initial classes and five added per task, could the model preserve earlier-task performance while learning new classes poorly? Please clarify the Table 3 results with accuracies by task or class group at the end of the sequence.
-
What do the opposite effects of dissimilar task ordering in Table 7 tell us about when FreeMOCA works reliably? What evidence supports the explanation in Appendix B.4?
-
Are the SGD results in Tables 2 and 9 directly comparable, and what explains the difference between 65.2% and 69.09%? Are the FreeMOCA results in Tables 2 and 7 from the same runs?
-
Minor wording clarification. Does “persistent variance repair” in the last sentence of Section 5.2 (page 7) refer to a repair step or to the variance collapse in Table 1?
4 comments
Request for feedback
Dear Reviewer,
We have tried our best to address the concerns. Please let us know if you have any further concerns. We will do our best to address those. Your response will be greatly appreciated.
Thanks
Thank the authors for the detailed rebuttal and the additional clarifications provided in the other discussion threads. The new old/new-class performance analyses directly address my concerns about retention and the stability–plasticity trade-off. The additional CKA measurements also provide useful evidence for the proposed explanation of representation compatibility across tasks.
I also appreciate the clarification that the primary contribution lies in studying low-loss connectivity along continual learning trajectories rather than proposing a novel interpolation algorithm. However, I remain somewhat unconvinced that the geometric findings offer sufficiently substantial new insights beyond existing studies of shared initialization, optimization history, and mode connectivity. While the additional evidence strengthens the empirical analysis, the incremental nature of the main contribution remains my primary concern.
Overall, the rebuttal has addressed several of my concerns, but I am inclined to maintain my original rating.
Novelty Beyond Prior Mode-Connectivity Studies
Thank you for clarifying the remaining concern. We respectfully believe our contribution is not a combination of previously established observations, but a systematic investigation of the factors that make direct linear connectivity possible, useful, or unreliable between consecutive CL states.
The prior studies differ in their objectives, compared solutions, and explanations. Frankle et al. (2020) study stability to SGD noise by comparing models optimized on the same objective (same training task), not successive CL tasks with changing objectives. Mirzadeh et al. (2021) investigate connectivity between continual and multitask (Joint) solutions, rather than between immediately consecutive CL states. They also examine CKA, but as a similarity measure between solutions, not to analyze evolving representation compatibility and its relationship to task-order sensitivity. Wen et al. (2023) investigate adjacent CL minima, but primarily characterize interpolation under different classifier-initialization schemes and address high-loss ridges by optimizing nonlinear paths and replay. CLeWI (2024) interpolates consecutive models but relies on replay-based neuron matching and BatchNorm recalibration. None establishes the combined conditions under which direct, replay-free consolidation is viable.
Our experiments provide four complementary insights:
-
Separating the factors behind connectivity. We distinguish independent initialization, shared ancestry with separate optimization, and continuous warm-starting. On EMBER/AZ, warm-started models are only 3.4%/1.1% closer than shared-initialization models, yet barriers decrease by 48.6%/58.1% and normalized barriers by 44.9%/57.1%. Thus, the additional connectivity associated with sequential optimization cannot be explained by parameter proximity alone.
-
Representation reuse as a contributing factor. Beyond initialization and optimization history, we investigate how compatibility of learned representations relates to connectivity. Block2 CKA increases from 0.553→0.885 (EMBER) and 0.496→0.928 (AZ) across early and late transitions. Under dissimilar ordering, EMBER CKA falls 0.677→0.436, barriers increase 66.2%, and accuracy drops 65.2%→22.86%; AZ maintains similar CKA (0.755→0.742) and improves in accuracy. These contrasting outcomes identify representation compatibility as an important empirical condition, not a guaranteed causal mechanism.
-
When are neuron alignment and variance repair necessary? CLeWI performs explicit neuron permutation and BatchNorm recalibration using stored examples. Its own no-rehearsal ablation still requires a memory buffer for these operations and reports poor results. We investigate a different regime: consecutive models inherit parameter correspondence through warm-started training, potentially eliminating explicit neuron matching. We further measure activation-variance contraction rather than assuming recalibration is required. Significant contraction occurs in only 1/10 EMBER and 4/10 AZ transitions, with none after the fifth. Thus, direct interpolation can succeed without replay-based alignment or normalization repair in the settings studied, although this is not guaranteed under larger representation shifts.
-
Connecting geometry to consolidation. Unlike methods that optimize alternative connectivity paths or rely on replay, we demonstrate how the existing linear path can support earlier-task retention while preserving substantial new-task plasticity. FreeMOCA is deliberately a minimal application of this finding, not a novel interpolation operator.
The central insight is therefore not merely that low-loss connectivity exists, but which properties of sequential learning support it, when additional alignment or repair can be avoided, and when representational changes undermine its usefulness. We believe this controlled characterization and its practical consequences constitute a distinct empirical contribution beyond the cited studies.
Please let us know whether this concern is addressed or if you have any further concerns. We will try our best to further clarify.
Clarifying Paper’s Contribution: Retention, Stability–Plasticity, and the Conditions for Low-Loss Connectivity
Thank you for the detailed questions. We agree that simply removing replay, alignment, and repair from an existing interpolation method is not enough to establish novelty. Our central contribution is the study of the low-loss paths themselves; FreeMOCA is a minimal use of that observation.
Please see 1, 2 and 4 in Reviewer NJwe.
Retention versus learning. We agree that Figure 2's consolidated segment is part of the adaptation line, not independent evidence of retention, and that low barriers alone do not guarantee preservation of earlier tasks. We therefore compare the previous → adapted → consolidated endpoints directly. At task 4→5, old-class accuracy changes from 52.28→21.83→46.43% on EMBER and 50.31→23.76→47.92% on AZ, while new-class accuracy changes from 13.54→91.16→85.22% and 17.06→88.45→80.27%, respectively. Corresponding old-task losses are 2.35→4.20→2.76 (EMBER) and 2.52→3.96→2.63 (AZ). At the final transition, old-class accuracy changes from 15.37→8.23→20.81% (EMBER) and 20.51→16.49→25.67% (AZ), with new-class accuracy of 9.38→83.54→72.66% and 6.14→92.31→69.05%, respectively. Old-task losses decrease after consolidation from 5.52 to 4.41 on EMBER and from 5.06 to 4.39 on AZ. These results show that adaptation learns new classes at the expense of earlier performance, while consolidation reduces old-task loss and recovers earlier accuracy, retaining substantial new-class performance. They also reveal a stability–plasticity trade-off, especially at the final transition. We clarify that Eq. (15) provides a conditional fixed-endpoint bound, with the classifier-expansion contribution explicitly separated but not bounded.
Does FreeMOCA neglect the newest classes? No. At the final transition, FreeMOCA's new-class accuracy before consolidation is 83.54% for EMBER and 92.31% for AZ while after consolidation is 72.66% vs 85.11% for fine-tuning (EMBER) and 69.05% vs 78.85% (AZ); retention is not obtained by refusing to learn new classes. Across available transitions, newly introduced-class accuracy averages 84.39% on EMBER and 87.52% on AZ. These numbers show that the method retains some older performance without simply ignoring new classes; they do not imply forgetting is eliminated. We have reported task by task accuracies at the end of sequence in Table 4 and Section 5.4.
Task ordering and conditions. The opposite effects reveal an important condition for FreeMOCA: tasks that are distant in input space do not necessarily require incompatible learned representations. Under dissimilar ordering, accuracy drops from 65.2% to 22.86% on EMBER, but increases from 63.7% to 67.86% on AZ. On EMBER, Block2 CKA decreases substantially (0.677→0.436), accompanied by increased parameter displacement (3.42→4.10, +19.9%) and interpolation barriers (0.0231→0.0384, +66.2%); the distance-normalized barrier also rises 15.7%. In contrast, AZ Block2 CKA barely changes (0.755→0.742), suggesting that its internal representations remain largely compatible despite greater input-space separation. This supports our explanation in Appendix B.4: FreeMOCA works more reliably when successive tasks can reuse learned features, maintaining favorable interpolation geometry. EMBER's deterioration is consistent with greater representational changes, while AZ retains substantial similarity. See 2 in Reviewer NJwe.
Experimental reporting. Tables 2 and 7(9 in the new version) use fixed λ=0.6; Table 9 used adaptive layer-wise λ and should not have been called comparable. We have replaced the corrected results in Table 11.
Wording and scope. “Persistent variance repair” was imprecise: we meant the activation-variance contraction in Table 1, not a repair procedure. FreeMOCA has no explicit alignment, BN recalibration, or variance repair. We have edited the paper to prevent further confusion.
Please let us know if the responses addresses your concerns. If you have any further concerns, please let us know, we will do our best to address those.