- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors of the paper operate on rehearsal-free class-incremental learning with a frozen ViT-B/16 and a growing pool of LoRA experts. They argue that routers that normalize scores over the expert set (softmax and similar) shift every old gate when a new expert is added. Gates that each depend only on their own expert (“factorized” gates) do not shift, and therefore propose the method ZDEP.
Critical strengths
Clear design principle. Showing that renormalization over a growing expert set rescales old gates is simple and easy to check. Building the method around frozen, factorized gates is principled and easy to implement.
Practical properties. No exemplars, frozen backbone, about 2M trainable parameters, a simple growth rule, and code is promised.
Critical weaknesses
The theoretical results look mostly trivial to me, and the main “if and only if” claim is too strong.
Proposition 1: This is just a basic mathematical result. Dividing by a larger sum reduces the old gates' values. The claim that the result is “tight when all old scores are equal” is confusing, because it describes equality rather than a bound. Also, Sparsemax is listed as a normalized method, but it does not follow the formula used in the proposition. Therefore, the proposition does not cover Sparsemax as written.
Proposition 2: This result is almost self-evident. If a gate does not depend on other experts, adding new experts will not affect it.
Proposition 3: The argument is circular. The old parameters are frozen, and the method is required to remain unchanged whenever a new expert is added. Therefore, the conclusion that the gates cannot depend on other experts is already implied by the assumptions.
Proposition 4: The reasoning is unclear and inconsistent. It first claims that the gradients are zero for all parameters, but then says routing only works if the key is frozen.
The abstract also contradicts the theoretical analysis. It claims that freezing the keys means the highest-scoring gate remains unchanged. This is incorrect because adding new gates can change which gate has the highest score. Section 3.1 also acknowledges that a new gate can outperform the previous winner.
The paper does not measure its main claim. The experiments do not report routing accuracy, how often old samples are assigned to different experts after new experts are added, or how many experts the controller actually selects for each dataset.
There may also be label or task-ID leakage during inference. The abstract and Section 1 state that the method commits to a winning expert for each class after every task and reuses this assignment during inference. If test-time routing relies on these class-specific assignments, the method may require the test sample's class or task label. This would give the method access to information that may not be available in a real-world setting.
Overall, the paper is poorly presented and unfortunately needs significant improvement in both presentation and technical content. The writing and overall presentation could be more polished and structured, as the current version makes it difficult to follow the main contributions. The experimental evaluation would benefit from stronger and more comprehensive baselines, as well as a more carefully designed experimental setup. Finally, the theoretical analysis needs to be strengthened to provide more meaningful insights and better support the paper's central claims.
Questions for authors
None.