acceptodds
Desk rejected at ICLR 2027

Continual Learning Trajectory as Memory for Replay-Free Consolidation in Malicious Code Analysis

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Malware classifiers must continually adapt as new threats emerge, but sequential updates can overwrite knowledge of earlier malware families, leading to catastrophic forgetting. Continual learning (CL) methods address this problem, but most treat past knowledge as something that must be explicitly stored or reconstructed. In malware analysis, these approaches are undesirable because of storage and computational costs, data-access policies, and security concerns. This leads to a fundamental question: \em Can the CL trajectory itself act as a form of memory? We study this question through the geometry of CL. Our experiments reveal a clear progression in linear connectivity—independently trained models are separated by high-loss regions; shared initialization introduces partial connectivity; and warm-start CL strengthens this connectivity further by combining common parameter ancestry with a continuous optimization trajectory. Based on this finding, we propose FreeMOCA, a replay-free CL method that recursively consolidates consecutive model states through linear parameter interpolation to reinforce favorable geometry across tasks. Unlike approaches that rely on replay, optimize nonlinear connectivity, or align and repair models before merging, FreeMOCA exploits connectivity already present in the learning trajectory and strengthens it through recursive consolidation.. We evaluate FreeMOCA in class-incremental learning setting on large-scale Windows and Android malware benchmarks against 10 CL baselines. FreeMOCA achieves average accuracies of 65.2% on EMBER-Class and 63.7% on AZ-Class, outperforming the strongest evaluated CL baselines by 2.1 and 8.8 percentage points, respectively, while reducing forgetting to 0.006 and 0.000. It requires only 15.98 MB of constant cross-task memory and averages 38.05 s of training time per task.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
malware analysis, continual learning, catastrophic forgetting, linear mode connectivity, parameter-space geometry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ttQrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes FreeMOCA for class-incremental malware family classification without retaining past samples. The method adapts the previous consolidated model to a new task, then interpolates the parameters shared by the previous and adapted models. It copies the adapted parameters for new classifier outputs and uses the consolidated model to initialize the next task. The authors examine interpolation geometry, expand the recursive update, and derive a local loss bound to explain how the trajectory carries historical information. Experiments on Windows and Android malware benchmarks report competitive accuracy and retention with low training time and persistent storage requirements.

Critical strengths
  1. Practical problem and simple method. The replay-free setting addresses limits on retaining historical data when updating malware classifiers. FreeMOCA uses direct post-task interpolation without neuron alignment or variance repair.
  2. Performance and efficiency. In the main experiments, FreeMOCA outperforms the evaluated replay-free methods on both malware benchmarks and is competitive with replay-based approaches. Its training time and average persistent storage use are lower than those of the three replay-based methods in Table 4.
Critical weaknesses
  1. Research contribution. FreeMOCA's update to shared parameters and the exponentially weighted expansion in Proposition 2 have the same form as those in Lookahead [1]. CLeWI also uses recursive post-task interpolation [2]. The paper does not clearly explain what new insight comes from removing replay, alignment, and repair and evaluating the simplified method on malware benchmarks.

  2. Explanation of retention. The Consolidated segment in Figure 2 covers only the first λ fraction of the adaptation path. A lower barrier does not by itself mean lower loss on earlier tasks. The bound reduction in Eq. (14) assumes a fixed adapted endpoint, and the classifier-expansion term is not quantified. Figures 2–3 do not directly test how these analyses relate to the reported retention gains.

  3. Task ordering and applicability. Compared with randomized ordering, dissimilar ordering in Table 7 lowers average accuracy from 65.2% to 22.86% on EMBER-Class but raises it from 63.7% to 67.86% on AZ-Class. Appendix B.4 attributes this contrast to feature reuse, parameter displacement, and interpolation barriers. It reports no corresponding measurements or baseline results under the same ordering, leaving the cause of the drop and the conditions for reliable consolidation unclear.

  4. Experimental reporting. Table 2 reports 65.2±0.05% with default SGD, whereas Table 9 reports 69.09% for SGD and 64.19%–69.21% across initializations. The paper does not clarify whether the training settings and reporting protocols are comparable. The results for randomized task orders in Table 7 also exactly match Table 2's means and standard deviations, but it is unclear whether they come from the same runs.

[1] Lookahead Optimizer: k steps forward, 1 step back. NeurIPS 2019.

[2] Continual Learning with Weight Interpolation. CVPR 2024.

Questions for authors
  1. CLeWI reports that interpolation has limited effectiveness without replay during training [2]. Which conditions make FreeMOCA effective on these malware tasks, and what new evidence supports that explanation?

  2. Does interpolation improve earlier-task retention, and how does it affect new-task learning? Please compare earlier-task losses for both endpoints and the consolidated model in Figure 2, and report earlier-task and new-task accuracies separately in Figure 3.

  3. With 50 initial classes and five added per task, could the model preserve earlier-task performance while learning new classes poorly? Please clarify the Table 3 results with accuracies by task or class group at the end of the sequence.

  4. What do the opposite effects of dissimilar task ordering in Table 7 tell us about when FreeMOCA works reliably? What evidence supports the explanation in Appendix B.4?

  5. Are the SGD results in Tables 2 and 9 directly comparable, and what explains the difference between 65.2% and 69.09%? Are the FreeMOCA results in Tables 2 and 7 from the same runs?

  6. Minor wording clarification. Does “persistent variance repair” in the last sentence of Section 5.2 (page 7) refer to a repair step or to the variance collapse in Table 1?

4 comments
AuthorsOpenReview ↗

Clarifying Paper’s Contribution: Retention, Stability–Plasticity, and the Conditions for Low-Loss Connectivity

Thank you for the detailed questions. We agree that simply removing replay, alignment, and repair from an existing interpolation method is not enough to establish novelty. Our central contribution is the study of the low-loss paths themselves; FreeMOCA is a minimal use of that observation.

  1. Please see 1, 2 and 4 in Reviewer NJwe.

  2. Retention versus learning. We agree that Figure 2's consolidated segment is part of the adaptation line, not independent evidence of retention, and that low barriers alone do not guarantee preservation of earlier tasks. We therefore compare the previous → adapted → consolidated endpoints directly. At task 4→5, old-class accuracy changes from 52.28→21.83→46.43% on EMBER and 50.31→23.76→47.92% on AZ, while new-class accuracy changes from 13.54→91.16→85.22% and 17.06→88.45→80.27%, respectively. Corresponding old-task losses are 2.35→4.20→2.76 (EMBER) and 2.52→3.96→2.63 (AZ). At the final transition, old-class accuracy changes from 15.37→8.23→20.81% (EMBER) and 20.51→16.49→25.67% (AZ), with new-class accuracy of 9.38→83.54→72.66% and 6.14→92.31→69.05%, respectively. Old-task losses decrease after consolidation from 5.52 to 4.41 on EMBER and from 5.06 to 4.39 on AZ. These results show that adaptation learns new classes at the expense of earlier performance, while consolidation reduces old-task loss and recovers earlier accuracy, retaining substantial new-class performance. They also reveal a stability–plasticity trade-off, especially at the final transition. We clarify that Eq. (15) provides a conditional fixed-endpoint bound, with the classifier-expansion contribution explicitly separated but not bounded.

  3. Does FreeMOCA neglect the newest classes? No. At the final transition, FreeMOCA's new-class accuracy before consolidation is 83.54% for EMBER and 92.31% for AZ while after consolidation is 72.66% vs 85.11% for fine-tuning (EMBER) and 69.05% vs 78.85% (AZ); retention is not obtained by refusing to learn new classes. Across available transitions, newly introduced-class accuracy averages 84.39% on EMBER and 87.52% on AZ. These numbers show that the method retains some older performance without simply ignoring new classes; they do not imply forgetting is eliminated. We have reported task by task accuracies at the end of sequence in Table 4 and Section 5.4.

  4. Task ordering and conditions. The opposite effects reveal an important condition for FreeMOCA: tasks that are distant in input space do not necessarily require incompatible learned representations. Under dissimilar ordering, accuracy drops from 65.2% to 22.86% on EMBER, but increases from 63.7% to 67.86% on AZ. On EMBER, Block2 CKA decreases substantially (0.677→0.436), accompanied by increased parameter displacement (3.42→4.10, +19.9%) and interpolation barriers (0.0231→0.0384, +66.2%); the distance-normalized barrier also rises 15.7%. In contrast, AZ Block2 CKA barely changes (0.755→0.742), suggesting that its internal representations remain largely compatible despite greater input-space separation. This supports our explanation in Appendix B.4: FreeMOCA works more reliably when successive tasks can reuse learned features, maintaining favorable interpolation geometry. EMBER's deterioration is consistent with greater representational changes, while AZ retains substantial similarity. See 2 in Reviewer NJwe.

  5. Experimental reporting. Tables 2 and 7(9 in the new version) use fixed λ=0.6; Table 9 used adaptive layer-wise λ and should not have been called comparable. We have replaced the corrected results in Table 11.

  6. Wording and scope. “Persistent variance repair” was imprecise: we meant the activation-variance contraction in Table 1, not a repair procedure. FreeMOCA has no explicit alignment, BN recalibration, or variance repair. We have edited the paper to prevent further confusion.

Please let us know if the responses addresses your concerns. If you have any further concerns, please let us know, we will do our best to address those.

AuthorsOpenReview ↗

Request for feedback

Dear Reviewer,

We have tried our best to address the concerns. Please let us know if you have any further concerns. We will do our best to address those. Your response will be greatly appreciated.

Thanks

Thank the authors for the detailed rebuttal and the additional clarifications provided in the other discussion threads. The new old/new-class performance analyses directly address my concerns about retention and the stability–plasticity trade-off. The additional CKA measurements also provide useful evidence for the proposed explanation of representation compatibility across tasks.

I also appreciate the clarification that the primary contribution lies in studying low-loss connectivity along continual learning trajectories rather than proposing a novel interpolation algorithm. However, I remain somewhat unconvinced that the geometric findings offer sufficiently substantial new insights beyond existing studies of shared initialization, optimization history, and mode connectivity. While the additional evidence strengthens the empirical analysis, the incremental nature of the main contribution remains my primary concern.

Overall, the rebuttal has addressed several of my concerns, but I am inclined to maintain my original rating.

AuthorsOpenReview ↗

Novelty Beyond Prior Mode-Connectivity Studies

Thank you for clarifying the remaining concern. We respectfully believe our contribution is not a combination of previously established observations, but a systematic investigation of the factors that make direct linear connectivity possible, useful, or unreliable between consecutive CL states.

The prior studies differ in their objectives, compared solutions, and explanations. Frankle et al. (2020) study stability to SGD noise by comparing models optimized on the same objective (same training task), not successive CL tasks with changing objectives. Mirzadeh et al. (2021) investigate connectivity between continual and multitask (Joint) solutions, rather than between immediately consecutive CL states. They also examine CKA, but as a similarity measure between solutions, not to analyze evolving representation compatibility and its relationship to task-order sensitivity. Wen et al. (2023) investigate adjacent CL minima, but primarily characterize interpolation under different classifier-initialization schemes and address high-loss ridges by optimizing nonlinear paths and replay. CLeWI (2024) interpolates consecutive models but relies on replay-based neuron matching and BatchNorm recalibration. None establishes the combined conditions under which direct, replay-free consolidation is viable.

Our experiments provide four complementary insights:

  1. Separating the factors behind connectivity. We distinguish independent initialization, shared ancestry with separate optimization, and continuous warm-starting. On EMBER/AZ, warm-started models are only 3.4%/1.1% closer than shared-initialization models, yet barriers decrease by 48.6%/58.1% and normalized barriers by 44.9%/57.1%. Thus, the additional connectivity associated with sequential optimization cannot be explained by parameter proximity alone.

  2. Representation reuse as a contributing factor. Beyond initialization and optimization history, we investigate how compatibility of learned representations relates to connectivity. Block2 CKA increases from 0.553→0.885 (EMBER) and 0.496→0.928 (AZ) across early and late transitions. Under dissimilar ordering, EMBER CKA falls 0.677→0.436, barriers increase 66.2%, and accuracy drops 65.2%→22.86%; AZ maintains similar CKA (0.755→0.742) and improves in accuracy. These contrasting outcomes identify representation compatibility as an important empirical condition, not a guaranteed causal mechanism.

  3. When are neuron alignment and variance repair necessary? CLeWI performs explicit neuron permutation and BatchNorm recalibration using stored examples. Its own no-rehearsal ablation still requires a memory buffer for these operations and reports poor results. We investigate a different regime: consecutive models inherit parameter correspondence through warm-started training, potentially eliminating explicit neuron matching. We further measure activation-variance contraction rather than assuming recalibration is required. Significant contraction occurs in only 1/10 EMBER and 4/10 AZ transitions, with none after the fifth. Thus, direct interpolation can succeed without replay-based alignment or normalization repair in the settings studied, although this is not guaranteed under larger representation shifts.

  4. Connecting geometry to consolidation. Unlike methods that optimize alternative connectivity paths or rely on replay, we demonstrate how the existing linear path can support earlier-task retention while preserving substantial new-task plasticity. FreeMOCA is deliberately a minimal application of this finding, not a novel interpolation operator.

The central insight is therefore not merely that low-loss connectivity exists, but which properties of sequential learning support it, when additional alignment or repair can be avoided, and when representational changes undermine its usefulness. We believe this controlled characterization and its practical consequences constitute a distinct empirical contribution beyond the cited studies.

Please let us know whether this concern is addressed or if you have any further concerns. We will try our best to further clarify.

Reviewer_NJweOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual learning for malware classification tasks. The paper proposes FreeMOCA, a replay-free method that directly interpolates the updated parameters with the previous parameters. The paper also analyzes the loss bounds and geometry of FreeMOCA theoretically. The paper shows that such a simple interpolation method can achieve good results compared to baselines on two malware classification task benchmarks.

Critical strengths
  1. The paper shows that a surprisingly simple baseline that interpolates the updated parameters with the old parameters can achieve very good performance on continual learning tasks for malware classification. Particularly, this method does not require replay training.
Critical weaknesses
  1. The scope is quite limited. The paper particularly focuses on continual learning for malware classification. However, the paper does not show which specific properties of this task could contribute to the success of the proposed simple method. Therefore, it is worth checking if this method can work on more general continual learning tasks.

  2. The theoretical analysis is useful, but it does not bring much insight. The propositions are more or less standard analysis involving standard interpolation-error bound, algebraic expansion of an exponential average, Taylor expansion and a norm bound. It does not give a convincing explanation on why such a simple interpolation method can work so well.

  3. lambda=0.9 achieves very good performance. This says that when heavily biased towards the new task, the performance becomes very good. On the other hand, this raises questions about what the performance would be for lambda=0.95, 0.98, etc. On the other hand, this is also slightly against the proposed idea: a weak interpolation with the old parameters achieves good performance.

  4. The proposed method intervenes with the learning rate or optimizer design. More analysis about various learning rate should be provided. A lookahead optimizer also has a similar idea of interpolation.

Questions for authors

Please address the weaknesses.

2 comments
AuthorsOpenReview ↗

Beyond Weight Interpolation: Understanding Low-Loss Connectivity Along Continual Learning Trajectories

Thank you for the feedback. We agree interpolation and standard bounds alone would not be a scientific contribution.

  1. What is new? Our main observation is that interpolation and low-loss connectivity are not the same thing. Any two parameter vectors can be interpolated, but the networks between them may have high loss, even if both endpoints perform well on their respective tasks. CLeWI uses interpolation between consecutive CL models but does not investigate why a low-loss path should exist between them, or under what conditions an intermediate state can provide a good balance between stability and plasticity. We study a different question from how to average weights: do low-loss linear paths exist between immediate CL solutions, why do they exist, and how do they change as tasks accumulate? Building on prior findings in linear connectivity to shared optimization history (Frankle et al., 2020), common initialization (Mirzadeh et al., 2021), and its evolution across CL transitions (Wen et al., 2023), we separate two effects that are usually mixed together. With independent initialization, the mean barriers are 0.0664 (EMBER) and 0.0567 (AZ); with shared initialization and separate training, 0.0496/0.0427; with consecutive warm-starting, 0.0255/0.0179. Shared ancestry already helps; continued optimization on the same trajectory lowers the barriers further. This is an empirical explanation, not a mathematical theorem. They open several research directions: using trajectory geometry to guide adaptive consolidation, detecting potentially conflicting task transitions through changes in connectivity, and designing training procedures that preserve compatibility between successive models, potentially enabling more effective CL and adapter merging. A broader question is whether we can deliberately shape the geometry of sequentially trained models to improve future learning, consolidation, and reuse. FreeMOCA is the simplest application we tested, not our claim to invent interpolation. We updated the Introduction and Conclusion.

  2. Why can direct interpolation work here? Malware tasks can reuse executable structures, API/permission information, and static code features, allowing successive models to refine existing representations. We examine this using Centered Kernel Alignment (CKA), measuring hidden representation similarity on the same inputs (higher values indicate greater similarity). Between consecutive consolidated and adapted states, mean Block2 CKA increases from 0.553→0.885 (EMBER) and 0.496→0.928 (AZ) when comparing the first four and last six transitions. On EMBER, this increase persists on a fixed initial-class probe (0.598→0.927), confirming that changing test composition alone does not explain the trend. Under dissimilar order, CKA drops substantially on EMBER (0.677→0.436) but barely changes on AZ (0.755→0.742), consistent with their contrasting accuracy trends. These findings suggest increasing representation reuse, although the trend is not monotonic. Unlike independently initialized models, warm-started states inherit parameter correspondence, potentially making CLeWI's neuron alignment unnecessary. Our variance analysis also finds severe contraction mainly in early transitions, suggesting BN repair may become less necessary later. Together, these observations explain why direct interpolation can work when successive representations remain compatible, without guaranteeing low-loss connectivity or retention. For broader CL benchmarks, see 7. Scope (reviewer 37VA).

  3. Theory and λ. The propositions use standard tools and are intended to frame, not establish, the finding. Proposition 1 links the barrier to displacement and curvature; Proposition 2 is a recency-weighted identity, not proof of memory; Proposition 3 gives a conditional old-task bound, not guaranteed retention or plasticity. At λ=0.9, EMBER accuracy is 62.6%, versus 15.4% for λ=1 and 65.2% at our default λ=0.6. A 10% old-state contribution is small at one transition, but it is applied after every task and changes the initialization for the next task. Thus λ=0.9 is not one weak interpolation performed at the end. Interpolation with λ=0.95/0.98 reached 25.04% and 24.63%, respectively. We edited Section 3.3 and A.4.

  4. Optimizer and Lookahead. Lookahead interpolates slow and fast weights after every k optimizer steps within the same training task, resetting fast weights to stabilize optimization and reduce variance.FreeMOCA perform post-task interpolation to balance stability and plasticity. It does not modify gradient updates or within-task learning rates, but changes the initialization of later tasks. Unlike Lookahead, FreeMOCA also operates with an expanding classifier and without replay. We do not claim novelty in the interpolation algebra or removing CLeWI's components.

Please let us know if the responses addresses your concerns. If you have any further concerns, please let us know, we will do our best to address those.

AuthorsOpenReview ↗

Request for feedback

Dear Reviewer,

We have tried our best to address the concerns. Please let us know if you have any further concerns. We will do our best to address those. Your response will be greatly appreciated.

Thanks

Reviewer_37VAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies replay-free class-incremental learning for malware classification. It argues that consecutive warm-started CL solutions are linearly connected, with lower barriers than shared- or independently-initialized solutions. It proposes FreeMOCA, which interpolates the previous consolidated model with the newly adapted mode after each task. It reports large gains over 10 baselines on EMBER-Class and AZ-Class, near-zero forgetting, and low time and memory cost.

Critical strengths
  1. One line of parameter interpolation, no replay buffer, no generator, no alignment. Training cost is close to plain fine-tuning, which suits the deployment constraints of malware pipelines.
  2. The authors explicitly state that the propositions do not guarantee zero forgetting, that connectivity ≠ no forgetting, and report a failure case.
  3. Two datasets, 10 baselines, multiple metrics, memory/time costs, variance-collapse diagnostics, block-wise ablations, and architecture/optimizer variations. The CLeWI ablation is informative.
Critical weaknesses
  1. Prop. 2 is misleading as "memory". With λ=0.6, task 1's weight is 0.4^{t−1} (≈1e-4 by task 10), so this is recency-weighted averaging. Also, the θ^a_k are not independent: each depends on θ^c_{k−1}. Prop. 3 only shows that moving less lowers an upper bound on old-task loss, which is trivial. It says nothing about the cost to new-task performance (plasticity). The Hessian-bounded-along-segment assumption is also dubious for ReLU networks, which are not C².
  2. By Prop. 1, the barrier scales with ‖d_t‖². Warm-started pairs are simply closer in parameter space than independently trained ones, so lower barriers may reflect small displacement rather than special "connectivity". No displacement-normalized analysis is given. Loss barriers on average are small in absolute terms, and no accuracy-along-path curves are shown.
  3. Table 8 labels 66.30/66.10 as "FreeMOCA" while the main FreeMOCA is 65.2/63.7. Adaptive ranges behave erratically: [0.3,0.7] gives 66.3, but [0.4,0.6] gives 40.5 and [0.1,0.9] gives 27.9, even though [0.5,0.5] gives 64.9. This non-monotonic collapse contradicts claims of robustness.
  4. Final evaluation uses the consolidated model, which is never directly trained on the latest task. The paper gives no accuracy matrix, per-task new/old accuracy, or intransigence/learning-accuracy analysis.
  5. The paper does not state how BN running statistics are interpolated or whether new-class logit bias is handled.
  6. The method is generic, yet only one MNIST sanity check is given (61.5% for a 5-task split, far below typical accuracies, compared against a single baseline). No other domains or real temporal-drift malware benchmarks (e.g., time-split data) are used.
Questions for authors
  1. Why are the std values in Table 2 (≈0.05) and Table 7 so small, while Table 9 shows a ≈5-point spread across seeds? What exactly varies across the 3 seeds (init, data order, class order)?
  2. How were F and BWT computed? For LwF and CLeWI, F < |BWT|, which contradicts standard definitions.
  3. How are BN running statistics and new-class logits handled during interpolation? Please add head-only, BN-only, and classifier-bias-correction controls to explain the jump from 15.4% (λ=1) to 62.6% (λ=0.9).
  4. How was λ chosen on a "validation split" without access to old-task data?
  5. Can you evaluate on real time-split malware drift datasets and on more standard CL benchmarks?
4 comments
AuthorsOpenReview ↗

Clarifying the Theoretical Claims, Experimental Consistency, and Scope of Trajectory-Based Connectivity

Thank you for the careful review. We clarify seven points.

  1. Connectivity and theory. Our empirical contribution is investigating whether low-loss linear paths exist between immediate CL solutions, what creates them, and how they evolve(see 1, 2 Reviewer NJwe). Proposition 1 gives a conditional upper bound, not proof of connectivity. Proposition 2 is recency-weighted but its coefficients describe direct endpoint contributions, not retained task knowledge. Adapted states are dependent, each starting from the preceding consolidation. We revised it and its proof in A.4. Proposition 3 conditionally bounds old-task loss under local smoothness; it guarantees neither new-task plasticity nor applicability across ReLU boundaries. We revised it and its proof in A.5.
  2. Is connectivity only proximity? Proximity matters but is insufficient. For independent/shared-initialization/warm-start pairs, distances are 26.50/6.53/6.31 (EMBER) and 26.54/6.72/6.65 (AZ), with barriers 0.0664/0.0496/0.0255 and 0.0567/0.0427/0.0179, respectively. Barrier/||d||² ratios are 9.46e-5/1.16e-3/6.40e-4 (EMBER) and 8.05e-5/9.45e-4/4.05e-4 (AZ). Warm-start distances are only 3.4%/1.1% smaller than shared-initialization distances, yet barriers fall 48.6%/58.1% and normalized barriers 44.9%/57.1%. Thus similarly distant solutions have different barriers. Under dissimilar EMBER ordering, displacement increases 3.42→4.10 (+19.9%), barriers 0.0231→0.0384 (+66.2%), the aggregate distance-normalized barrier rises 15.7%, and Block2 CKA decreases 0.677→0.436. Proximity contributes, but representational changes also accompany differences in geometry.
  3. Inconsistent tables. Tables 2 and 7 (now 9) use fixed λ=0.6; old Table 9 used adaptive layer-wise λ and was not matched. We replaced its results in Table 11. Table 8 (now 10) reports an adaptive variant (66.30/66.10), not the fixed-λ default (65.2/63.7). Some adaptive ranges are sensitive, whereas fixed λ is stable across ordinary randomized orders.
  4. F, BWT, REM. Following Díaz-Rodríguez et al. (2018), we define trajectory-averaged BWT as BWT=2T(T−1)∑i=2T∑j=1i−1(Ri,j−Rj,j).\mathrm{BWT}=\frac{2}{T(T-1)}\sum_{i=2}^{T}\sum_{j=1}^{i-1}(R_{i,j}-R_{j,j}). Forgetting FF instead averages max⁡k≥jRk,j−RT,j\max_{k\ge j}R_{k,j}-R_{T,j} over earlier tasks. Thus, BWT measures average intermediate performance changes, whereas FF measures final drops from historical best performance; then, FF need not exceed ∣BWT∣|\mathrm{BWT}|. We define REM=1−∣min⁡(BWT,0)∣\mathrm{REM}=1-|\min(\mathrm{BWT},0)|, not 1−F1-F. Thus, CLeWI on AZ can have BWT = +0.0144, REM = 1.0, and F = 0.0589.
  5. BatchNorm, classifier, and plasticity. λ=0.9 outperforms λ=1 because consolidation occurs after every task, changing subsequent warm starts, not merely the final checkpoint. Shared backbone/BN parameters and running statistics are interpolated without recalibration. Head expansion preserves old outputs, reinitializes output-BN statistics, and uses the adapted expanded head with consolidated features. On EMBER at λ=0.9: fine-tuning=15.4%, full=62.6%, backbone-only=59.8%, BN-only=23.3%, head-only=31.72%, and fine-tuning+bias correction=19.85%. Backbone consolidation explains most of the improvement. At final transition, FreeMOCA new-class accuracy before consolidation is 83.54% (EMBER) and 92.31% (AZ); after consolidation it is 72.66% versus 85.11% for fine-tuning (EMBER), and 69.05% versus 78.85% (AZ). Retention does not mean refusing to learn new classes.
  6. λ selection without replay. λ=0.6 was selected offline using only the initial 50 classes: a simulated 20+6×5 sequence with 10% validation, then fixed for all 11 real tasks. No past samples are used during consolidation. We clarified Section 6.
  7. Scope. Chronological EMBER/AZ Domain-IL improves over fine-tuning by 5.4/1.2 points and exceeds EWC, SI, and MalCL. See Table 8 and section B.4. On ResNet-18, CIFAR-10 achieves 52.16% versus 23.34% fine-tuning (joint 93.59%); CIFAR-100 achieves 39.04% versus 10.12% (joint 72.42%). The gains over the None suggest CL trajectory can act as a limited form of implicit memory without replay, rather than establishing parity with stronger replay-based methods. Our finding suggest this property is especially useful when successive tasks share reusable representations, as in structured, feature-based malware datasets, rather than being specific to malware. Connectivity barriers for independent/shared-initialization/warm-start/FreeMOCA are 0.49/0.37/0.29/0.25 (MNIST), 0.72/0.43/0.36/0.34 (CIFAR-10), and 1.58/1.12/0.96/0.97 (CIFAR-100). Shared ancestry and warm-starting reduce barriers on all three; FreeMOCA further lowers them on MNIST/CIFAR-10, but remains comparable to warm-start on CIFAR-100. The higher CIFAR-100 barriers highlight limitations when tasks require greater representational change: low-loss connectivity is setting-dependent, not guaranteed.

Please let us know if the responses addresses your concerns. If you have any further concerns, please let us know, we will do our best to address those.

AuthorsOpenReview ↗

Request for feedback

Dear Reviewer,

We have tried our best to address the concerns. Please let us know if you have any further concerns. We will do our best to address those. Your response will be greatly appreciated.

Thanks

Thank you for the rebuttal. I appreciate the effort to clarify several important points, especially the interpolation details, metric definitions, and the added normalized-barrier analysis. However, the theoretical issues are addressed mostly by weakening/reframing the claims rather than by materially strengthening the theory. So the theory now feels more like intuitive support than strong formal justification. I also appreciate the added results beyond malware, but the cross-domain validation is still limited on standard continual learning benchmarks, so I think the scope of the claims should remain cautious. Overall, I tend to keep my score.

AuthorsOpenReview ↗

Clarifying the Empirical Contribution: Theoretical Rigor, Cross-Domain Validation, and Scientific Significance

Thank you for reconsidering our rebuttal and acknowledging the additional analyses. We respectfully believe two important distinctions deserve further consideration:

1. Theoretical clarification versus theoretical weakness. We respectfully disagree that clarifying the scope of our propositions weakens the contribution. The purpose of our theoretical analysis is to provide a mathematical interpretation of the observed behavior, not to establish a new theorem guaranteeing linear connectivity. Our research question is empirical: when do consecutive CL solutions become linearly connected, what factors contribute to this geometry, and under what conditions does it deteriorate? We revised the propositions to state their assumptions and conclusions precisely, directly addressing the soundness concerns raised in the original review. Making a mathematical statement more precise should not be regarded as a weaker contribution simply because it does not establish a stronger theorem.

More importantly, our central findings do not depend on these propositions. We investigate the geometry directly through controlled experiments that separate the effects of shared initialization, parameter proximity, and continuous optimization. On EMBER/AZ, warm-started solutions are only 3.4%/1.1% closer than shared-initialization solutions, yet their barriers decrease by 48.6%/58.1%, with normalized barriers decreasing by 44.9%/57.1%. These results show that parameter proximity alone is insufficient to explain the observed connectivity. We further examine how representation reuse, activation-variance stability, and task ordering relate to this geometry, and directly measure its consequences for old-task retention and new-task learning. These findings are supported by experimental evidence independently of the theoretical bounds.

We therefore believe the relevant question is not whether the revised propositions constitute a stronger theoretical contribution, but whether the empirical investigation provides new and meaningful insights into continual learning. Our experiments identify contributing factors, examine competing explanations, and reveal conditions under which direct interpolation becomes less reliable. This is the scientific contribution we ask the reviewer to evaluate.

2. Cross-domain validation and the scope of our claims. Our paper focuses on continual learning for malicious code analysis, as stated in its title. Malware classification presents its own challenges: new malware families continually emerge, retaining historical samples and continuous retraining may not be feasible due to storage and computational costs, data-access policies or sample-sharing constraints while models need to adapt efficiently without losing earlier knowledge. We believe that understanding and addressing these challenges is a worthwhile research contribution in its own right. A study focused on improving continual learning for malware analysis should not have to demonstrate effectiveness in every other domain to establish its value. Nevertheless, we have now evaluated our method on five datasets, extending beyond the two malware datasets to MNIST, CIFAR-10, and CIFAR-100, including ResNet-18 experiments. This is already broader than the evaluation commonly used in continual learning, where many papers report results on only two or three datasets, often limited to MNIST and CIFAR variants. Thus, evaluating two malware datasets alongside standard continual learning benchmarks should not be interpreted as evidence of insufficient empirical validation. The independent → shared-initialization → warm-start connectivity trend persists across these benchmarks, although barriers and consolidation effectiveness vary substantially. Importantly, CIFAR-100 reveals a limitation rather than uniformly favorable results: its higher barriers and weaker consolidation benefits demonstrate that this geometry is task-dependent. We report these limitations precisely because identifying when an empirical phenomenon fails is part of understanding it.

ICLR's reviewing guidelines recognize that new, relevant, and impactful empirical knowledge can constitute a significant contribution without requiring state-of-the-art results or a novel theoretical framework. We respectfully believe our revised work should be assessed in this context.

We would appreciate clarification on whether any specific technical concern remains unresolved, beyond the desire for stronger theoretical guarantees or broader validation. Given the additional controlled evidence, revised theoretical claims, evaluation on five datasets spanning malware and standard continual learning benchmarks, and explicit limitations, we respectfully ask you to reconsider whether the original assessment still reflects the scientific contribution of the revised paper.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.