- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes continual learning benchmarks derived from coarse and fine category splits of MMMU and MMStar, asserting that existing benchmarks suffer from trivial task separability. It introduces CoMem, a training-free retrieval method for vision-language models. The method does not update model weights and does not require task identity at inference. On the proposed benchmarks, it obtains accuracy comparable to or slightly better than CL-MoE
Critical strengths
The paper identifies an important limitation of existing continual learning benchmarks. High task separability may help expert routing without requiring meaningful knowledge transfer. The proposed intra-dataset task streams offer a alternative benchmark.
Critical weaknesses
- I am unclear about the contribution and positioning of the paper. The proposed benchmark is constructed by splitting only two existing datasets using predefined subject/category labels. There is no new dataset or guidelines defines. Also, task order, task difficulty, semantic similarity between tasks, and cross-task transfer are not studied.
- The motivation is high task separability of existing benchmarks. But, reducing nearest-prototype task-identification accuracy from approximately 100% to 60–77% does not establish that the resulting benchmark measures meaningful knowledge transfer.
- Although the paper cites recent multimodal continual learning approaches, only CL-MoE is evaluated. And the proposed retrieval-based method is not compared against simple kNN memory, retrieval-based in-context learning, or other training-free alternatives.
- All experiments use only one split and a fixed task order.
- Major numerical inconsistencies across all experimental tables. several results are inconsistent across tables 1, 3, 4 and 5
- Paper needs major revision. Figure1, and 2 have same captions and there are many typos, Table numbers references are wron
Questions for authors
- How does the proposed benchmark measure continual knowledge transfer rather than sequential adaptation to different subject categories? What is the proposes formulation or guideline?
- How does CoMEM compare against simple nearest-neighbor answer retrieval or retrieval-based in-context learning under the same memory budget?
- How sensitive are the results to task ordering, task difficulty, and semantic overlap between categories?
- What evidence shows that CoMEM acquires transferable knowledge rather than primarily retrieving memorized answer information?
- Tables 1, 3, 4, and 5 report different CoMEM accuracies under apparently identical settings? Which values are correct?