- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SCALE, a continual VLM learning framework that monitors predefined foundation skills and regenerates verified supervision when skill performance degrades. It combines skill maintenance with current-task SFT through task-prioritized gradient projection, without replaying historical downstream-task examples. Experiments on UCo-VQA with Molmo2-4B report improved final accuracy and reduced forgetting.
Critical strengths
-
The paper provides a meaningful perspective on continual learning by focusing on reusable pretrained capabilities rather than only historical task knowledge.
-
SCALE improves final accuracy over continual SFT under both SS and PS settings, with additional analyses of supervision generation, maintenance budgets, and task-order sensitivity.
Critical weaknesses
-
The connection between foundation-skill degradation and downstream forgetting is not directly established. Figure 4 lacks a no-maintenance comparison, while the near-zero pretrained scores of Count and NLI raise questions about whether these skills are preserved or newly acquired.
-
The key components, including drift-aware selection, budget allocation, and gradient projection, are not independently validated. Evaluation is also limited to one VLM backbone, leaving generalizability uncertain.
Questions for authors
-
How does the drift gate operate for skills with near-zero pretrained scores, particularly Count (Table 5)? Can the authors report unrounded scores and actual activation statistics?
-
Can the authors provide budget-matched ablations for drift-based skill selection/allocation and task-priority gradient projection?
-
Can the authors compare skill evolution under SCALE and continual SFT on independent skill evaluation data, and establish its relationship with downstream forgetting?
-
The experiments are limited to Molmo2-4B. How well would SCALE generalize to other VLM backbones, particularly when comparable pretraining data or skill annotations are unavailable?
8 comments
Response to Reviewer LgCm [part 2]
[W2 & Q2] Ablation of Core Components
i) Core-component ablations. To evaluate SCALE's two core mechanisms, we conduct two ablations (Table 3), i.e., w/o drift-aware gate (uniform allocation), which disables drift-aware skill selection and allocates the maintenance budget uniformly across all eight foundation skills, and w/o task-priority projection, which directly combines task and skill gradients without projection. All other training settings remain unchanged.
Table 3: Component ablations on VQA-v3 under SS (%).
| Method | Loc. | Comm. | Type | Act. | Color | Count | Recog. | Judge | FAA↑ | CAA↑ | FFM↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SCALE (Full) | 56.05 | 92.07 | 33.46 | 68.96 | 67.94 | 63.23 | 35.78 | 90.74 | 63.53 | 62.46 | 2.91 |
| w/o drift-aware gate (uniform allocation) | 57.19 | 90.11 | 30.38 | 68.65 | 37.06 | 42.02 | 35.68 | 90.55 | 56.46 | 60.50 | 10.43 |
| w/o task-priority projection | 56.20 | 92.02 | 33.16 | 68.63 | 64.91 | 63.25 | 35.40 | 90.53 | 63.01 | 61.22 | 2.86 |
As shown in Table 3, replacing drift-aware gating and allocation with uniform maintenance reduces FAA and CAA by 7.07 and 1.96 points, respectively, while increasing FFM by 7.52 points, demonstrating the importance of allocating maintenance resources according to foundation-skill degradation. Removing task-priority projection decreases FAA and CAA by 0.52 and 1.24 points, respectively, despite a marginally lower FFM (2.86 vs. 2.91). These results support the contribution of task-priority projection to a more favorable stability–plasticity trade-off, rather than minimizing forgetting alone. See Table 4 of the revised manuscript for the component-ablation results.
ii) Cross-backbone applicability. Regarding the single-backbone limitation, we additionally evaluate SCALE on SmolVLM-Base, as detailed in our response to Q4. SCALE improves FAA from 36.94% to 38.21% and reduces FFM from 12.30% to 9.27%, providing further evidence of cross-backbone applicability. The corresponding comparison is provided in Table 2 of the revised manuscript.
[Q3] Foundation-Skill Evolution and Its Relationship to Downstream Forgetting
i) Foundation-skill functionality. To examine whether foundation-skill maintenance preserves reusable capabilities during continual adaptation, we compare the final functionality of all eight skills under SCALE and continual SFT after the complete eight-task sequence.
Table 4: Final foundation-skill functionality after eight downstream tasks (%).
| Foundation Skill | Continual SFT | SCALE | Gain (pp) |
|---|---|---|---|
| Point | 10.65 | 55.23 | +44.58 |
| Count | 54.16 | 54.78 | +0.62 |
| Attribute/Color | 64.36 | 66.31 | +1.95 |
| NLI | 34.30 | 37.72 | +3.42 |
| Commonsense Bool | 52.81 | 65.73 | +12.92 |
| Commonsense MCQ | 67.91 | 68.66 | +0.75 |
| Sensemaking | 52.27 | 57.98 | +5.71 |
| Visual Commonsense | 92.92 | 93.55 | +0.63 |
As shown in Table 4, SCALE achieves higher final functionality than continual SFT across all eight skills, with notable gains in Point (+44.58 pp), Commonsense Bool (+12.92 pp) and Sensemaking (+5.71 pp).
ii) Downstream retention. The maintenance records and individual skill-removal ablations in W1 & Q1 show that all eight foundation skills receive maintenance and that removing any individual skill decreases FAA and increases FFM. Together with the functionality comparison in Table 4, these results further support that supervising and maintaining evolving foundation skills helps mitigate downstream forgetting. Consistently, SCALE reduces final downstream FFM from 11.99 under continual SFT to 2.91. See Tables 1 and 9 of the revised manuscript for the downstream and foundation-skill results.
Response to Reviewer LgCm [part 3]
[Q4] Generalization to Other VLM Backbones
To evaluate the generalizability of SCALE beyond Molmo2-4B, we conduct additional experiments on SmolVLM-Base, another pretrained VLM with publicly available training data. We instantiate the same eight foundation-skill categories using model-specific pretraining-related data while retaining SCALE's core mechanisms and the original UCo-VQA VQA-v3 benchmark under SS.
Table 5: Cross-backbone continual-learning performance on VQA-v3 under SS (%).
| Method | Loc. | Comm. | Type | Act. | Color | Count | Recog. | Judge | FAA↑ | CAA↑ | FFM↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|
| SmolVLM SFT | 39.93 | 73.79 | 16.42 | 59.50 | 8.18 | 0.70 | 21.45 | 75.58 | 36.94 | 41.40 | 12.30 |
| SmolVLM SCALE | 42.61 | 76.21 | 16.71 | 60.27 | 8.86 | 1.62 | 22.84 | 76.53 | 38.21 | 43.76 | 9.27 |
As shown in Table 5, SCALE improves FAA by 1.27 percentage points and CAA by 2.36 points, while reducing FFM by 3.03 points compared with continual SFT. Notably, SCALE achieves higher final accuracy across all eight downstream tasks, providing additional evidence that foundation-skill maintenance can benefit continual adaptation across different pretrained VLM backbones, rather than relying exclusively on Molmo2-specific capabilities. See Table 2 of the revised manuscript for the cross-backbone evaluation.
Response to Reviewer LgCm [part 1]
We thank the reviewer for the careful and constructive suggestions. We have conducted additional experiments to address the concerns and incorporated the corresponding clarifications and revisions into the revised manuscript.
[W1 & Q1] Foundation-Skill Functionality, Gate Activation, and Downstream Retention
i) Skill Activation and Individual Contributions
To examine whether the selected foundation skills are actually maintained during continual learning and contribute to downstream performance, we record their maintenance-budget allocations across downstream stages and conduct leave-one-skill-out ablations for all eight skills.
Table 1: Foundation-skill maintenance allocations across downstream stages (numbers in parentheses denote the cumulative budget allocated to each skill across maintenance refreshes within that stage).
Table 2: Leave-one-skill-out ablations on VQA-v3 under SS (%).
Table 1 confirms that all eight foundation skills receive maintenance allocations during continual learning, including Count and NLI. Table 2 further shows that removing any individual skill reduces FAA and increases FFM relative to full SCALE, although the magnitude varies across skills. These results support the positive contributions of the selected foundation-skill maintenance objectives to downstream learning and retention. See Tables 7 and 8 of the revised manuscript for the activation and removal results.
ii) Clarification of Initial Skill Scores and Drift-Aware Gate Activation
We thank the reviewer for noting the unexpectedly low initial foundation-skill scores in the original Table 5. The pretrained Count score was mistakenly reported as 0.000, and the correct offline probe score is 0.281.
To enable efficient and repeated skill monitoring during continual adaptation, SCALE employs constrained short-answer generation, essential-answer extraction, and skill-specific normalized matching. Both the pretrained and continually adapted models are evaluated under the same online protocol, ensuring consistent measurement of functionality degradation relative to each skill's pretrained reference. In contrast, offline evaluation aims to quantify absolute skill functionality by directly comparing model predictions with the original ground-truth (GT) annotations. Differences between probe and pretraining distributions, as well as prediction–annotation format mismatches, can lead to low scores under strict matching. Such offline scores therefore do not necessarily reflect the absence of reusable foundation capabilities.
Gate activation is based on relative online functionality degradation rather than absolute offline scores. At each maintenance refresh, SCALE measures degradation against the fixed pretrained online reference, applies EMA smoothing, and activates skills whose smoothed degradation exceeds (τref=0.05). The fixed maintenance budget is then dynamically allocated among activated skills according to their measured degradation. Thus, correcting the offline Count score does not affect the online reference or the Gate decisions made during training. The corresponding mechanism is detailed in Eqs. (2)–(6) of the revised manuscript.