- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a replay-free, persistently adaptive framework for visual language models called DualMem-Evo, which combines evolutionary semantic memory with task-specific contextual experts. The authors introduce prototype-guided expert retrieval and low-rank integration to reuse historical knowledge, and incorporate memory separation and semantic stability regularization.
Critical strengths
-
The evaluation covers heterogeneous datasets, different task orders, limited supervision, unidentified module selection, and class-incremental learning.
-
Architecture ablation experiments use comparable LoRA overall rankings, which helps distinguish the impact of memory organization from the impact of increased adapter capacity.
Critical weaknesses
-
Robustness of the principal gains has not been confirmed. Under Order-I conditions, the reported improvements compared to DPW were approximately 0.3 and 0.7 percentage points in Average and Last, respectively; improvements were also generally small under other MTIL settings. Similarly, adding PGEEM to controlled dual memory only improved Average from 79.36 to 79.59 and Last from 88.78 to 88.95, equivalent to only 0.23 and 0.17 percentage points of gain, respectively. The paper does not report the training seed mean or variability for these principal comparisons, making it difficult to determine whether the observed differences reflect consistent gains or normal fluctuations during training runs. This is particularly important in component ablation experiments, as small incremental gains are used to support the contribution of PGEEM. The authors should replicate principal baseline comparisons and related ablation experiments under matched experimental conditions using multiple independent training seeds and report the mean and standard deviation or confidence intervals. Reporting variability in performance differences will help further determine whether the proposed method consistently outperforms baseline methods and whether PGEEM offers reproducible additional advantages.
-
Appendix A.3 describes scoring candidate objects using normalized prediction confidence and image-text consistency, followed by candidate selection and temperature-weighted fusion. However, this description does not fully clarify the specific scoring functions, normalization operations, number of candidate objects, temperature, or parameter selection procedures. For example, it is unclear how these two scoring terms are calculated and combined, on which candidate objects or sample sets the normalization is performed, and how many candidate objects are evaluated and retained. The mathematical relationship between candidate object scores and final fusion weights also needs to be clarified. These choices directly determine which candidate objects contribute to the prediction and their weights, so different reasonable implementations may produce different results. Authors should provide complete formulas or pseudocode, along with the numerical settings used in the experiments. They should also explain how these settings were chosen, what data were used for selection, and whether the same settings were shared across different datasets and task sequences. A focused sensitivity analysis of the number of candidates and fusion temperature helps assess the dependence of the reported gains on these choices.
Questions for authors
-
Could you provide the complete equations or pseudocode for the inference process in Appendix A.3, including all candidate scores, selection rules, fusion parameters, and their tuning schemes? Could you compare the complete model and the model containing only episodic memories under the same inference rules to differentiate between training gains and candidate fusion gains?
-
After completing T tasks, are all T integrated episodic memory experts retained? If so, the ranking of each expert is bounded, but the total storage capacity still increases with T. What storage object does the compression ratio in Appendix B refer to? Please specify the total storage capacity and the number of uppropagation steps or latency during inference.
16 comments
2-1Response to Reviewer xFdH
2. Complete Inference Equations in Appendix A.3, Leak-Free Selection Protocol, and Hyperparameter Sensitivity Analysis (W2, Q1a)
2.1 Complete Inference Scoring, Normalization, and Temperature-Scaled Fusion Equations
Under standard MTIL (Order-I, Order-II, 5-shot, 16-shot), seen tasks are evaluated directly using the current semantic memory paired with the corresponding episodic expert ; for unseen future tasks (as well as unknown/ambiguous samples in IFMS-MTIL), the model performs label-free selection and fusion over the candidate pool formed by the learned expert set (each combined with the current ) plus frozen CLIP.
For an individual test sample and target class set (of size ) specified by the evaluation protocol, each candidate model extracts normalized visual features and target class text features to compute cosine similarity logits and class probability distributions: Note: All normalizations are performed independently within each single test sample across target classes or across candidate experts , with zero reliance on statistics from other samples in the test batch. We evaluate the reliability of candidate on sample along two complementary dimensions:
- Normalized Prediction Confidence: Derived from the normalized Shannon entropy of the class distribution as the confidence term :
- Cross-Modal Image–Text Consistency: Defined as the maximum cosine similarity between the sample's visual feature and the best-matching class text feature:
- Joint Reliability Scoring and Top- Temperature-Scaled Fusion: Multiplying the two terms yields the candidate reliability score , requiring a high-scoring candidate to simultaneously exhibit "low predictive uncertainty (sharp distribution)" and "high cross-modal semantic alignment": After computing for all candidates, we retain at most the Top- highest-scoring candidates to form the set , and map them via fusion temperature into final fusion weights : (Note: For the complete equations of IFMS-MTIL empirical quantile compatibility routing and CIL hierarchical owned-class margin inference, as well as the full list of training/evaluation hyperparameters across all experimental tables, please see our Response to Reviewer ELGv, Sections 3.2 and 3.3.)
2-2&3Response to Reviewer xFdH
2. Complete Inference Equations in Appendix A.3, Leak-Free Selection Protocol, and Hyperparameter Sensitivity Analysis (W2, Q1a)
2.2 Strict Selection Protocol for Parameters and Inference Configurations (Data Provenance & Selection Protocol)
In response to your concern regarding "how these hyperparameters were selected, what data were used, and whether they are shared across datasets," we explicitly clarify: the selection of all model parameters and inference configurations strictly follows the six-stage train/validation isolation protocol below, without ever using any test-set samples or test results for tuning:
| Stage | Data Used | Action & Strict Criterion |
|---|---|---|
| 1. Data Split | Official training set of currently accessible tasks | Partition into a Train split and a Validation split, and save sample indices |
| 2. Model Training | Train split only | Optimize dual-memory LoRA parameters, layer-wise collaboration coefficients , and trainable weights |
| 3. Config Selection | Validation split only | Evaluate on the validation split to select fusion temperature , candidate count , routing thresholds, etc. |
| 4. Config Freezing | Zero reference to any test results | Freeze the final selected global configuration and save the configuration alongside its validation selection basis |
| 5. Final Evaluation | Official test set | Compute the full accuracy matrix and Transfer / Average / Last metrics once under the frozen configuration |
| 6. Multi-Seed Verification | Fixed protocol & frozen configuration | Vary only the training random seed, reporting mean, standard deviation, and paired gains |
Cross-Task and Cross-Sequence Sharing: Once determined on the validation split, and serve as globally frozen hyperparameters shared identically across all datasets and all sessions in Order-I, Order-II, 5-shot, 16-shot, and IFMS-MTIL, without any per-dataset or per-order adjustment.
2.3 Sensitivity Analysis on Retained Candidate Count and Fusion Temperature
Following your suggestion, we conducted dedicated sensitivity sweeps over the retained candidate count (on full Order-I) and the fusion temperature (across the first 5 sessions):
| Retained Candidates After Scoring | 1 (Top-1 Hard Routing) | 3 | 5 (Default) | 7 | 9 |
|---|---|---|---|---|---|
| Transfer (%) (Order-I) | 70.03 | 70.26 | 70.37 | 70.46 | 70.49 |
| Fusion Temperature | 0.01 | 0.05 | 0.1 (Default) | 0.2 | 0.5 |
|---|---|---|---|---|---|
| Transfer (%) (5-Session) | 64.68 | 64.40 | 64.45 | 64.34 | 64.31 |
The experiments demonstrate that: (i) compared to hard single-expert selection (, 70.03%), multi-candidate soft fusion () effectively mitigates the risk of single-expert misselection and maintains smooth gains across (with default balancing accuracy and computational efficiency); and (ii) as the fusion temperature varies across a range , Transfer fluctuates by only 0.37% (and by only 0.14% within ), proving that our method is highly robust to inference hyperparameters and that its gains do not rely on delicate hyperparameter tuning.
1-Response to Reviewer xFdH
Response to Reviewer xFdH
We sincerely thank Reviewer xFdH for recognizing our comprehensive evaluation across multiple heterogeneous datasets, different task orders, few-shot regimes, identification-free module selection (IFMS-MTIL), and class-incremental learning (CIL), as well as our rank-matched (Comparable Overall Ranks) architectural ablation design (CS1, CS2). Your precise and incisive questions regarding the multi-seed statistical stability of our main and ablation gains, complete inference fusion equations and rigorous hyperparameter selection criteria, controlled architectural comparison under unified inference rules, and the exact scope of SVD compression and deployment overhead go straight to the heart of experimental rigor. Below, we address each of your points in detail.
1. Magnitude Analysis of Main & PGEEM Ablation Gains and Multi-Seed Verification (W1)
(1) Contextualizing the 0.3%–0.7% Gains on MTIL: Empirical Performance Ceiling and Cross-Protocol Consistency
On MTIL benchmarks, recent CLIP-based continual learning baselines (such as DPW and KeepLoRA+) already operate in a high-accuracy regime. To objectively quantify the remaining room for improvement, we computed an offline Empirical Ceiling reference on the Order-I accuracy matrix: for each unseen task, we take the highest transfer accuracy observed at any stage prior to learning that task; for each seen task, we replace its entry with the score of an independent per-task LoRA fine-tuning (Oracle LoRA Fine-tune) model. This yields a theoretical combined reference of 71.1% (Transfer) / 80.4% (Average) / 89.4% (Last).
Consequently, the strongest baseline DPW (70.4% / 79.3% / 88.3%) is only 0.7% / 1.1% / 1.1% away from this empirical ceiling. Within this highly saturated regime, DualMem-Evo's 70.4% / 79.6% / 89.0% effectively eliminates roughly 30%–64% of the remaining gap between the strongest baseline and the empirical ceiling. More importantly, this advantage does not occur by chance in a single setting, but holds consistently across Order-II (+0.1% / +0.4% / +0.5%), 5-shot (+0.0% / +0.2% / +0.5%), 16-shot (Average +0.4%, Last +0.5%), IFMS-MTIL (+0.3% / +0.4% / +0.3%), and all 6 CIL benchmarks (up to +2.27% in Average; see Appendix C.1; and +2.54% / +4.09% / +4.21% over CL-LoRA under matched settings; see our Response to Reviewer ELGv, Section 1).
(2) Multi-Seed Statistical Evaluation Under Strictly Matched Protocols (Multiple Independent Training Seeds)
Following your recommendation, under strictly matched experimental conditions, we repeated our experiments across multiple independent training random seeds for the strongest baseline DPW, our controlled dual-memory baseline (Regulated Dual Memory w/o PGEEM), and the full DualMem-Evo (w/ PGEEM), reporting means, standard deviations, and paired differences:
The multi-seed statistics clearly demonstrate that: (i) Full DualMem-Evo exhibits minimal sensitivity to training seed randomness (with standard deviations of only 0.02%, 0.01%, and 0.05% on Transfer, Average, and Last—substantially smaller than DPW's variance), reliably outperforming the strong baseline DPW; and (ii) incorporating PGEEM delivers stable, reproducible positive paired gains across all three metrics (+0.24% / +0.23% / +0.19%) while significantly reducing the standard deviation of final retention (Last) from 0.11% to 0.05%. Furthermore, in the newly added Appendix D.5 and our controlled retrieval ablation, we confirm that our Centered signed collaboration—designed in accordance with the inevitable positive/negative polarization of layer-wise weights—not only significantly accelerates early-to-mid training convergence and lowers cross-entropy loss during new-memory formation, but also consistently outperforms Nearest and Random retrieval across all final metrics (see our Response to Reviewer Ttpt, Section 3 for detailed analysis and comparison tables).