- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper develops a masked AdamW optimizer and a ledger that records changes to routed base parameters, including adapter merges. It uses this accounting tool to study a continual-learning controller that commits updates to base weights, defers them to replay-trained LoRA adapters, or withholds them. On a seven-phase Qwen3-0.6B distillation stream, a random-COMMIT policy with a similar aggregate commit rate gives comparable forgetting to the evidence-based controller. The paper concludes that the controller's selection policy adds no measurable benefit at this operating point, while substantially increasing write volume increases forgetting.
Critical strengths
- The masked optimizer and merge-aware ledger provide a concrete way to check which parameters actually change, rather than relying on the gate's intended decisions.
- The paper reports a negative result and distinguishes the comparisons that meet its equivalence criterion from those that remain unresolved.
- The random control matches several aggregate quantities, including commits, replay, effective writes, and realized drift. The paper also reports unresolved accounting discrepancies rather than hiding them.
Critical weaknesses
-
The random control does not isolate module selection from the other consequences of the policy. It matches commit rates at the phase level, but does not establish an exact match in commit timing, module identity, and replay history. Figure 9 shows that 89% of sampled controller commits occur in the first 200 steps of each phase, so timing may matter. The result supports the narrower conclusion that the full evidence policy did not outperform this simpler rate-matched policy. It does not by itself establish that choosing which modules to update is unimportant. A control that keeps the exact per-step number of commits and randomizes only module identity would make the interpretation stronger.
-
The broader significance of the negative result is unclear. The audited controller is the authors' own construction, combining several signals and thresholds. Showing that it does not beat a random policy is useful, but does not establish a limitation of optimizer-state gates generally. The paper acknowledges that its control cannot assess STABLE, and the theoretical limitation on zero-initialized optimizer-state gates is a separate result. I would like to see the accounting tool and matched controls applied to an established gate, or a clearer justification for why this particular controller is an informative test case. The related-work claim that prior gates lack volume-matched random controls should also be reconciled with the Random BCD comparison in MoFO.
-
The retention evaluation is narrow. The primary protocol uses one 0.6B student, three stream seeds, and 32 fixed teacher-forced validation rows per domain. Shared parameters remain trainable outside the routed gate and account for 26% of the model. Their reported drift is much larger than that of the routed parameters; this alone does not prove they dominate forgetting, but it makes a shared-frozen control important. The paper needs evidence that the null comparison persists when shared learning is controlled and when retention is measured on a larger held-out set or generated outputs.
Questions for authors
- Can you show the random arm's commit-time distribution and add a control that matches the controller's exact per-step number of commits while randomizing module identity? Please distinguish any remaining differences in replay and merge history.
- Why is the proposed controller the right target for the audit? Can the accounting tool be applied to an established gate, and how does the random-control contribution differ from MoFO's Random BCD comparison?
- Does the controller-versus-random result persist when shared parameters are frozen? Can the retention comparison be repeated on a larger evaluation set and with generation-based metrics?
- What caused the two unexplained coordinate changes in the zero-write audit? Can the authors localize and resolve them before making an implementation-level claim of exact isolation?