acceptodds
Under review as a conference paper at ICLR 2027

How Foundation Skills Boost Continual Vision-Language Learning: A Skill-Evolution Perspective

Abstract

Continual learning can empower large vision-language models (VLMs) to acquire new knowledge in non-stationary environments without replaying historical downstream-task data, yet continual post-training often causes catastrophic forgetting. We argue that such forgetting also stems from the degradation of reusable foundation capabilities inherited from pretraining. This paper therefore revisits a fundamental yet underexplored problem: how can foundation skills in VLMs boost continual vision-language learning performance? We propose a Skill-Centric Adaptation model for Lifelong Evolution, termed SCALE, which reinterprets pretrained VLMs as a repertoire of reusable foundation skills. As continual downstream tasks come, downstream task adaptation follows supervised fine-tuning (SFT), while SCALE adopts skill-specific drift-aware gates to identify drifting foundation skills and task-prioritized co-evolution to maintain them. These selected skills can evolve with the model through verifier-guided behavioral supervision, without constraining current-task adaptation. This co-evolution strategy allows foundation skills to better support new-task learning while mitigating forgetting associated with foundation-skill drift during task-biased post-training. A fixed maintenance budget is dynamically allocated across foundation skills according to capability drift, independent of downstream task history. We evaluate SCALE on VQA-v3 under the SS and PS settings of UCo-VQA, with analyses of continual-learning performance, generalization, sensitivity, ablations and foundation-skill evolution. Our SCALE improves final average accuracy by 8.7% and reduces final forgetting by 9.1% points over matched continual SFT.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, vision-language models, foundation skills

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LgCmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCALE, a continual VLM learning framework that monitors predefined foundation skills and regenerates verified supervision when skill performance degrades. It combines skill maintenance with current-task SFT through task-prioritized gradient projection, without replaying historical downstream-task examples. Experiments on UCo-VQA with Molmo2-4B report improved final accuracy and reduced forgetting.

Critical strengths
  1. The paper provides a meaningful perspective on continual learning by focusing on reusable pretrained capabilities rather than only historical task knowledge.

  2. SCALE improves final accuracy over continual SFT under both SS and PS settings, with additional analyses of supervision generation, maintenance budgets, and task-order sensitivity.

Critical weaknesses
  1. The connection between foundation-skill degradation and downstream forgetting is not directly established. Figure 4 lacks a no-maintenance comparison, while the near-zero pretrained scores of Count and NLI raise questions about whether these skills are preserved or newly acquired.

  2. The key components, including drift-aware selection, budget allocation, and gradient projection, are not independently validated. Evaluation is also limited to one VLM backbone, leaving generalizability uncertain.

Questions for authors
  1. How does the drift gate operate for skills with near-zero pretrained scores, particularly Count (Table 5)? Can the authors report unrounded scores and actual activation statistics?

  2. Can the authors provide budget-matched ablations for drift-based skill selection/allocation and task-priority gradient projection?

  3. Can the authors compare skill evolution under SCALE and continual SFT on independent skill evaluation data, and establish its relationship with downstream forgetting?

  4. The experiments are limited to Molmo2-4B. How well would SCALE generalize to other VLM backbones, particularly when comparable pretraining data or skill annotations are unavailable?

8 comments
AuthorsOpenReview ↗

Response to Reviewer LgCm [part 1]

We thank the reviewer for the careful and constructive suggestions. We have conducted additional experiments to address the concerns and incorporated the corresponding clarifications and revisions into the revised manuscript.

[W1 & Q1] Foundation-Skill Functionality, Gate Activation, and Downstream Retention

i) Skill Activation and Individual Contributions

To examine whether the selected foundation skills are actually maintained during continual learning and contribute to downstream performance, we record their maintenance-budget allocations across downstream stages and conduct leave-one-skill-out ablations for all eight skills.

Table 1: Foundation-skill maintenance allocations across downstream stages (numbers in parentheses denote the cumulative budget allocated to each skill across maintenance refreshes within that stage).

Downstream TaskFoundation Skills (Allocated Budget)
LocationPoint (48), NLI (16), Count (16), CS-Bool (4)
CommonsensePoint (32), NLI (16), Sensemaking (8)
TypePoint (16)
ActionPoint (16)
ColorAttribute/Color (16)
CountAttribute/Color (16)
RecognitionPoint (49), Attribute/Color (96), CS-MCQ (24), Visual-CS (7)
JudgePoint (128), Attribute/Color (80), CS-MCQ (32)

Table 2: Leave-one-skill-out ablations on VQA-v3 under SS (%).

MethodLoc.Comm.TypeAct.ColorCountRecog.JudgeFAA↑CAA↑FFM↓
SCALE (Full)56.0592.0733.4668.9667.9463.2335.7890.7463.5362.462.91
w/o Point56.9891.9631.5368.7362.4263.8135.3390.7462.6961.543.17
w/o Count57.1291.3230.5568.0167.4663.7335.9890.9763.1461.352.98
w/o Attribute/Color56.3090.8630.9668.5410.6758.1734.7990.8355.1460.8812.07
w/o NLI55.3890.5031.1470.1562.8058.1037.9090.7062.0861.813.82
w/o CS-Bool56.7991.5032.2368.9568.2461.5935.3190.8263.1862.873.31
w/o CS-MCQ56.5489.0532.6163.9864.9057.3334.2090.6761.1661.603.21
w/o Sensemaking56.3792.1731.7470.0960.9753.7836.5890.8061.5661.546.00
w/o Visual-CS55.7190.6232.3267.9764.8263.5635.8190.8962.7161.783.35

Table 1 confirms that all eight foundation skills receive maintenance allocations during continual learning, including Count and NLI. Table 2 further shows that removing any individual skill reduces FAA and increases FFM relative to full SCALE, although the magnitude varies across skills. These results support the positive contributions of the selected foundation-skill maintenance objectives to downstream learning and retention. See Tables 7 and 8 of the revised manuscript for the activation and removal results.

ii) Clarification of Initial Skill Scores and Drift-Aware Gate Activation

We thank the reviewer for noting the unexpectedly low initial foundation-skill scores in the original Table 5. The pretrained Count score was mistakenly reported as 0.000, and the correct offline probe score is 0.281.

To enable efficient and repeated skill monitoring during continual adaptation, SCALE employs constrained short-answer generation, essential-answer extraction, and skill-specific normalized matching. Both the pretrained and continually adapted models are evaluated under the same online protocol, ensuring consistent measurement of functionality degradation relative to each skill's pretrained reference. In contrast, offline evaluation aims to quantify absolute skill functionality by directly comparing model predictions with the original ground-truth (GT) annotations. Differences between probe and pretraining distributions, as well as prediction–annotation format mismatches, can lead to low scores under strict matching. Such offline scores therefore do not necessarily reflect the absence of reusable foundation capabilities.

Gate activation is based on relative online functionality degradation rather than absolute offline scores. At each maintenance refresh, SCALE measures degradation against the fixed pretrained online reference, applies EMA smoothing, and activates skills whose smoothed degradation exceeds (τref=0.05\tau_{\mathrm{ref}}=0.05). The fixed maintenance budget is then dynamically allocated among activated skills according to their measured degradation. Thus, correcting the offline Count score does not affect the online reference or the Gate decisions made during training. The corresponding mechanism is detailed in Eqs. (2)–(6) of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer LgCm [part 2]

[W2 & Q2] Ablation of Core Components

i) Core-component ablations. To evaluate SCALE's two core mechanisms, we conduct two ablations (Table 3), i.e., w/o drift-aware gate (uniform allocation), which disables drift-aware skill selection and allocates the maintenance budget uniformly across all eight foundation skills, and w/o task-priority projection, which directly combines task and skill gradients without projection. All other training settings remain unchanged.

Table 3: Component ablations on VQA-v3 under SS (%).

MethodLoc.Comm.TypeAct.ColorCountRecog.JudgeFAA↑CAA↑FFM↓
SCALE (Full)56.0592.0733.4668.9667.9463.2335.7890.7463.5362.462.91
w/o drift-aware gate (uniform allocation)57.1990.1130.3868.6537.0642.0235.6890.5556.4660.5010.43
w/o task-priority projection56.2092.0233.1668.6364.9163.2535.4090.5363.0161.222.86

As shown in Table 3, replacing drift-aware gating and allocation with uniform maintenance reduces FAA and CAA by 7.07 and 1.96 points, respectively, while increasing FFM by 7.52 points, demonstrating the importance of allocating maintenance resources according to foundation-skill degradation. Removing task-priority projection decreases FAA and CAA by 0.52 and 1.24 points, respectively, despite a marginally lower FFM (2.86 vs. 2.91). These results support the contribution of task-priority projection to a more favorable stability–plasticity trade-off, rather than minimizing forgetting alone. See Table 4 of the revised manuscript for the component-ablation results.

ii) Cross-backbone applicability. Regarding the single-backbone limitation, we additionally evaluate SCALE on SmolVLM-Base, as detailed in our response to Q4. SCALE improves FAA from 36.94% to 38.21% and reduces FFM from 12.30% to 9.27%, providing further evidence of cross-backbone applicability. The corresponding comparison is provided in Table 2 of the revised manuscript.

[Q3] Foundation-Skill Evolution and Its Relationship to Downstream Forgetting

i) Foundation-skill functionality. To examine whether foundation-skill maintenance preserves reusable capabilities during continual adaptation, we compare the final functionality of all eight skills under SCALE and continual SFT after the complete eight-task sequence.

Table 4: Final foundation-skill functionality after eight downstream tasks (%).

Foundation SkillContinual SFTSCALEGain (pp)
Point10.6555.23+44.58
Count54.1654.78+0.62
Attribute/Color64.3666.31+1.95
NLI34.3037.72+3.42
Commonsense Bool52.8165.73+12.92
Commonsense MCQ67.9168.66+0.75
Sensemaking52.2757.98+5.71
Visual Commonsense92.9293.55+0.63

As shown in Table 4, SCALE achieves higher final functionality than continual SFT across all eight skills, with notable gains in Point (+44.58 pp), Commonsense Bool (+12.92 pp) and Sensemaking (+5.71 pp).

ii) Downstream retention. The maintenance records and individual skill-removal ablations in W1 & Q1 show that all eight foundation skills receive maintenance and that removing any individual skill decreases FAA and increases FFM. Together with the functionality comparison in Table 4, these results further support that supervising and maintaining evolving foundation skills helps mitigate downstream forgetting. Consistently, SCALE reduces final downstream FFM from 11.99 under continual SFT to 2.91. See Tables 1 and 9 of the revised manuscript for the downstream and foundation-skill results.

AuthorsOpenReview ↗

Response to Reviewer LgCm [part 3]

[Q4] Generalization to Other VLM Backbones

To evaluate the generalizability of SCALE beyond Molmo2-4B, we conduct additional experiments on SmolVLM-Base, another pretrained VLM with publicly available training data. We instantiate the same eight foundation-skill categories using model-specific pretraining-related data while retaining SCALE's core mechanisms and the original UCo-VQA VQA-v3 benchmark under SS.

Table 5: Cross-backbone continual-learning performance on VQA-v3 under SS (%).

MethodLoc.Comm.TypeAct.ColorCountRecog.JudgeFAA↑CAA↑FFM↓
SmolVLM SFT39.9373.7916.4259.508.180.7021.4575.5836.9441.4012.30
SmolVLM SCALE42.6176.2116.7160.278.861.6222.8476.5338.2143.769.27

As shown in Table 5, SCALE improves FAA by 1.27 percentage points and CAA by 2.36 points, while reducing FFM by 3.03 points compared with continual SFT. Notably, SCALE achieves higher final accuracy across all eight downstream tasks, providing additional evidence that foundation-skill maintenance can benefit continual adaptation across different pretrained VLM backbones, rather than relying exclusively on Molmo2-specific capabilities. See Table 2 of the revised manuscript for the cross-backbone evaluation.

Reviewer_jUTWOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SCALE, a continual learning framework for VLMs that tracks and maintains foundation skills inherited from pretraining while learning new tasks. For plasticity, each incoming VQA task is learned with standard SFT on current-task data only. For stability, the method monitors eight predefined foundation skills (e.g., Point, Count, Attribute/Color, NLI) on held-out probes, activates the skills whose scores degrade relative to the pretrained model, and allocates a fixed maintenance budget across them in proportion to their drift. For each activated skill, the stage-entry checkpoint generates responses to skill prompts drawn from Molmo2 pretraining data, and only responses accepted by a skill-specific verifier are used as maintenance targets. The two objectives are combined with an asymmetric gradient projection that leaves the task gradient unchanged and removes only the conflicting component of the skill gradient. As I understand it, each foundation skill is effectively a predefined task, the verifier functions as a benchmark-style evaluation against ground-truth annotations, and the approach therefore amounts to a rehearsal of self-generated, verifier-filtered pretraining data. The method is evaluated on VQA-v3 under the SS and PS settings of UCo-VQA with Molmo2-4B, improving final average accuracy by 8.7pp and reducing forgetting by 9.1pp over continual SFT.

Critical strengths
  • Shifting the preservation target to foundation capabilities.

The paper proposes preserving capabilities inherited from pretraining, rather than only retaining knowledge of past downstream tasks, as a target for continual learning. This is a well-motivated perspective: continual post-training can erode general capabilities that both previous and future tasks rely on, and replay-free methods that focus solely on past task performance do not directly address this. Defining these capabilities independently of the downstream task history also allows the maintenance cost to remain fixed as the task stream grows.

Critical weaknesses

W1. The notion of “foundation skills” is not sufficiently established.

The paper selects task categories from the pretraining data and treats them as foundation skills, but does not sufficiently establish that these categories correspond to capabilities already acquired by the pretrained model or that they provide reusable support across downstream tasks. Several skills show near-zero pretrained scores (Count 0.000, NLI 0.017, CS Bool 0.169 in Table 5), and with m₀ = 0, the Count skill can never be activated under Eq. 2. Meanwhile, the gains over continual SFT are concentrated on tasks that closely overlap with specific skills (Color +48.9pp, Count +20.4pp on SS, with the other six tasks within about ±2pp), and the Attribute/Color skill is partly built from COCO/VQA color QA. The improvement may therefore reflect rehearsal of data closely related to specific downstream tasks rather than general, reusable support. This is consequential because the paper’s central claim rests on this notion.

W2. Key design choices and claims are not validated.

The ablations in Table 2 cover only how maintenance responses are generated. There is no ablation of the drift-aware gate (e.g., versus uniform allocation) or of the task-priority projection (e.g., versus a simple sum or symmetric projection), so the contribution of two of the three core components is unclear. The co-evolution claim in Fig. 4 and Table 5 is also not compared against continual SFT without maintenance, so the observed functionality gains cannot be distinguished from the effect of downstream training itself (e.g., the Count stage). Finally, while the fixed budget keeps maintenance cost constant, task-specific knowledge outside the predefined skills is not preserved, as seen in the collapses on PS (e.g., Commonsense 76.78 → 1.58). This is consequential because it leaves unclear which parts of the method are responsible for the reported gains and how the method would scale to longer task streams.

Questions for authors

Q1. By what criteria were the eight skills selected, and why do they include text-only reasoning skills (e.g., NLI) for continual VQA? Would the same categories apply to other pretrained models, or must the skills be redefined for each model’s pretraining data?

Q2. Which skills are activated during each downstream task? Given the overlap between some tasks and skills (e.g., Color), do the authors consider all skills equally important, and could a leave-one-skill-out ablation clarify each skill’s contribution? Relatedly, what do the edges among skills in Fig. 2 represent, given that the method treats skills independently?

Q3. How were the skill-specific evaluation functions and thresholds determined (e.g., the shared activation threshold τ_ref = 0.05 across skills with different metrics, and the Point verification threshold of 0.75), and were they tuned on the evaluation stream?

Q4. The maintenance budget is fixed per refresh (B = 32 alongside 128 current-task samples) and distributed only across the eight predefined skills, so knowledge outside these skills receives no maintenance. Is the severe forgetting on PS (e.g., Commonsense 76.78 → 1.58 after the Type stage, Location 62.05 → 19.89 at the end) attributable to such knowledge? As more skills become activated over longer streams, each skill’s share of B shrinks; how should B be set as the number of tasks or skills grows, given that the budget sensitivity in Table 4 is non-monotonic (24 / 32 / 40)?

Q5. What exactly is meant by “behavioral realization evolves while functionality is maintained”? How is the skill-specific distance D_k defined, and how do Fig. 4 and Table 5 look for continual SFT without maintenance?

7 comments
AuthorsOpenReview ↗

Response to Reviewer jUTW [part 1]

We appreciate the reviewer's careful reading and thought-provoking questions. We have supplemented the experiments and clarified the methodological choices and limitations in the revised manuscript.

[Q1] Foundation-Skill Selection and Cross-Backbone Generalization

i) Foundation-skill selection. We select the eight foundation skills based on (a) established VLM capability dimensions identified by benchmarks such as MME [1] and MMStar [2], (b) the foundational capabilities required by downstream tasks, and (c) the availability of pretraining-related data with verifiable annotations. Text-only reasoning skills are included because VQA requires not only visual perception but also language understanding, semantic reasoning, and commonsense inference. Their contributions are further examined through the skill activation statistics and individual skill-removal ablations discussed in our response to Q2.

ii) Cross-backbone generalization. To investigate whether the skill repertoire must be redefined for different pretrained models, we further evaluate SCALE on SmolVLM-Base using the same eight foundation-skill categories and the identical downstream benchmark, with skill data drawn from its publicly available training resources.

Table 1: Cross-backbone continual-learning performance on VQA-v3 under SS (%).

MethodLoc.Comm.TypeAct.ColorCountRecog.JudgeFAA↑CAA↑FFM↓
SmolVLM SFT39.9373.7916.4259.508.180.7021.4575.5836.9441.4012.30
SmolVLM SCALE42.6176.2116.7160.278.861.6222.8476.5338.2143.769.27

As shown in Table 1, SCALE improves FAA and CAA by 1.27 and 2.36 percentage points, respectively, while reducing FFM by 3.03 points, with improvements across all eight downstream tasks. These results demonstrate that SCALE's effectiveness is not restricted to Molmo2-4B and provide evidence that the same foundation-skill repertoire can support continual adaptation across different pretrained VLMs without requiring backbone-specific skill redefinition. Further details are provided in Table 2 of the revised manuscript.

[W1 & Q2] Foundation-Skill Competence, Activation, and Contributions

i) Pretrained Foundation-Skill Competence

We thank the reviewer for noting the unexpectedly low initial foundation-skill scores in Table 5 of the original manuscript (Table 9 in the revised manuscript). We confirmed that Count was mistakenly reported as 0.000; the correct offline probe score is 0.281.

To enable efficient online monitoring and automatic verification, SCALE uses constrained short-answer generation, essential-answer extraction, and normalized matching. The online Gate tracks each skill's functionality degradation relative to its pretrained reference under a consistent monitoring protocol, whereas Table 9 of the revised manuscript quantifies absolute skill functionality on specific probes by comparing model predictions against the original ground-truth (GT) annotations. Differences between the probe and pretraining distributions, as well as potential mismatches between output formats and original annotations, can affect strict-matching scores. Thus, low initial scores do not necessarily imply the absence of reusable foundation capabilities or negate their value for continual learning. To further examine skill-functionality retention, we compare the final functionality of all eight skills under SCALE and continual SFT (Table 2).

Table 2: Final foundation-skill functionality after eight downstream tasks (%).

Foundation SkillContinual SFTSCALEGain (pp)
Point10.6555.23+44.58
Count54.1654.78+0.62
Attribute/Color64.3666.31+1.95
NLI34.3037.72+3.42
Commonsense Bool52.8165.73+12.92
Commonsense MCQ67.9168.66+0.75
Sensemaking52.2757.98+5.71
Visual Commonsense92.9293.55+0.63

As shown in Table 2, SCALE achieves higher final functionality than continual SFT across all eight skills, with notable gains in Point (+44.58 pp), Commonsense Bool (+12.92 pp) and Sensemaking (+5.71 pp). See Table 9 of the revised manuscript for the corresponding functionality comparison.

ii) Skill Activation and Individual Contributions

To examine the contributions of the eight foundation skills, we record their maintenance allocations across downstream stages and conduct individual skill-removal ablations under the same continual-learning protocol.

AuthorsOpenReview ↗

Response to Reviewer jUTW [part 2]

Table 3: Foundation-skill maintenance allocations across downstream stages (numbers in parentheses denote the cumulative maintenance budget allocated to each skill across refreshes within that stage).

Downstream TaskFoundation Skills (Allocated Budget)
LocationPoint (48), NLI (16), Count (16), CS-Bool (4)
CommonsensePoint (32), NLI (16), Sensemaking (8)
TypePoint (16)
ActionPoint (16)
ColorAttribute/Color (16)
CountAttribute/Color (16)
RecognitionPoint (49), Attribute/Color (96), CS-MCQ (24), Visual-CS (7)
JudgePoint (128), Attribute/Color (80), CS-MCQ (32)

Table 4: Leave-one-skill-out ablations on VQA-v3 under SS (%).

MethodLoc.Comm.TypeAct.ColorCountRecog.JudgeFAA↑CAA↑FFM↓
SCALE (Full)56.0592.0733.4668.9667.9463.2335.7890.7463.5362.462.91
w/o Point56.9891.9631.5368.7362.4263.8135.3390.7462.6961.543.17
w/o Count57.1291.3230.5568.0167.4663.7335.9890.9763.1461.352.98
w/o Attribute/Color56.3090.8630.9668.5410.6758.1734.7990.8355.1460.8812.07
w/o NLI55.3890.5031.1470.1562.8058.1037.9090.7062.0861.813.82
w/o CS-Bool56.7991.5032.2368.9568.2461.5935.3190.8263.1862.873.31
w/o CS-MCQ56.5489.0532.6163.9864.9057.3334.2090.6761.1661.603.21
w/o Sensemaking56.3792.1731.7470.0960.9753.7836.5890.8061.5661.546.00
w/o Visual-CS55.7190.6232.3267.9764.8263.5635.8190.8962.7161.783.35

Table 3 confirms that all eight foundation skills receive maintenance allocations, while Table 4 shows that removing any individual skill reduces FAA relative to full SCALE. These results support the positive, though varying, contributions of all eight skill-maintenance objectives to continual-learning performance. See Tables 7 and 8 of the revised manuscript for the skill-activation and removal results.

iii) Task-Aligned Supervision and Data-Source Dependence

The gains under SS are concentrated on Color and Count, which also exhibit substantial forgetting in other methods. For example, MaDQ-QReplay [3] achieves final accuracies of 75.93% and 61.92% on these tasks, compared with 19.00% and 42.82% under continual SFT. Notably, SFT and SCALE perform similarly immediately after learning Color (76.70% vs. 76.22%), but differ substantially at the final stage (19.00% vs. 67.94%). This suggests that SCALE's improvement on Color arises primarily from better retention. We further examine task-aligned supervision through the ablations below.

Table 5: Task-aligned maintenance and data-source ablations under SS (%).

MethodLoc.Comm.TypeAct.ColorCountRecog.JudgeFAA↑CAA↑FFM↓
Continual SFT55.8391.0431.8670.6319.0042.8236.8990.7454.8559.9111.99
SCALE (Full)56.0592.0733.4668.9667.9463.2335.7890.7463.5362.462.91
w/o Color & Count PT56.5791.0832.4070.3027.6056.8036.3190.6757.7260.908.73
PixMo-only Color57.5890.5432.0470.6550.9668.5737.5390.4962.3061.574.28

First, removing both Color and Count foundation-skill maintenance still improves FAA by 2.87 points over continual SFT, with downstream Color and Count improvements of 8.60 and 13.98 points, respectively. This indicates that other maintained foundation skills can provide complementary support beyond directly corresponding task supervision.

Second, removing COCO/VQA-derived Color supervision and retaining only PixMo-based Color maintenance still achieves 62.30% FAA and 50.96% Color accuracy, outperforming continual SFT by 7.45 and 31.96 points, respectively. This directly addresses the concern that SCALE's Color improvement may be driven primarily by COCO/VQA-derived auxiliary supervision.

Together, these results indicate that SCALE's improvements cannot be explained solely by direct Color/Count maintenance or COCO/VQA-derived Color supervision, while supporting the role of foundation-skill maintenance in mitigating downstream forgetting.

AuthorsOpenReview ↗

Response to Reviewer jUTW [part 3]

iv) Clarification of Figure 2

The connections in Fig. 2 are intended to illustrate that multiple foundation skills may jointly influence the learning and performance of a downstream task. They serve as a conceptual illustration of this joint support rather than an explicitly modeled dependency structure. We have clarified this distinction in the caption of Fig. 2 in the revised manuscript.

[W2] Ablation of Key Components

We thank the reviewer for pointing out the missing component ablations. We conduct two ablations of SCALE's core components: w/o drift-aware gate (uniform allocation), which disables drift-aware skill selection and allocates the maintenance budget uniformly across all eight foundation skills, and w/o task-priority projection, which directly combines task and skill gradients without projection. All other training settings remain unchanged.

Table 6: Component ablations on VQA-v3 under SS (%).

MethodLoc.Comm.TypeAct.ColorCountRecog.JudgeFAA↑CAA↑FFM↓
SCALE (Full)56.0592.0733.4668.9667.9463.2335.7890.7463.5362.462.91
w/o drift-aware gate (uniform allocation)57.1990.1130.3868.6537.0642.0235.6890.5556.4660.5010.43
w/o task-priority projection56.2092.0233.1668.6364.9163.2535.4090.5363.0161.222.86

As shown in Table 6, replacing drift-aware gating with uniform budget allocation reduces FAA and CAA by 7.07 and 1.96 points, respectively, while increasing FFM by 7.52 points, demonstrating the importance of allocating maintenance resources according to foundation-skill degradation. Removing task-priority projection decreases FAA and CAA by 0.52 and 1.24 points, respectively, despite a marginally lower FFM (2.86 vs. 2.91). Thus, task-priority projection contributes to a more favorable stability–plasticity trade-off, rather than minimizing forgetting alone. See Table 4 of the revised manuscript for the component-ablation results.

[Q3] Skill-Specific Verification and Hyperparameter Sensitivity

To assess the sensitivity of the Point verification threshold, EMA smoothing coefficient, and Gate activation threshold, we conduct additional experiments on VQA-v3 under SS, as summarized below.

Table 7: Hyperparameter sensitivity on VQA-v3 under SS (%).

SettingLoc.Comm.TypeAct.ColorCountRecog.JudgeFAA↑CAA↑FFM↓
SCALE (Full)56.0592.0733.4668.9667.9463.2335.7890.7463.5362.462.91
Point Verifier = 0.5055.7092.3629.7768.6024.1252.8535.0090.4756.1160.3210.76
Point Verifier = 0.9056.3792.5031.3068.5263.6154.0835.9590.5061.6062.033.71
EMA β = 056.3888.2230.9068.5623.6053.9034.5890.4955.8359.358.47
EMA β = 0.955.8587.7331.7770.4464.2561.9637.6390.7462.5561.392.70
Gate τref = 0.0156.8492.1630.4468.6360.9959.4235.4590.5761.8161.603.46
Gate τref = 0.1056.4487.7732.0468.5065.7756.1435.6990.6661.6361.134.50

The default configuration uses a Point verification threshold of 0.75, EMA β=0.5\beta=0.5, and Gate activation threshold τref=0.05\tau_{\mathrm{ref}}=0.05.

i) Point Verification Threshold. The other seven foundation skills use essential-answer extraction and normalized matching, whereas Point predicts coordinate sets requiring tolerance for spatial deviations. We therefore adopt point-matching F1 with a separate verification threshold of 0.75. As shown in Table 7, this threshold outperforms 0.50 and 0.90 across FAA, CAA, and FFM.

ii) EMA Smoothing Coefficient. EMA with β=0.5\beta=0.5 stabilizes functionality-degradation estimates across maintenance refreshes. Removing smoothing (β=0\beta=0) substantially degrades performance, while stronger smoothing (β=0.9\beta=0.9) slightly improves FFM but reduces CAA from 62.46% to 61.39%. These results support β=0.5\beta=0.5 as a balanced choice.

iii) Gate Activation Threshold. All eight skills share τref=0.05\tau_{\mathrm{ref}}=0.05, which operates on functionality degradation relative to each skill's pretrained reference rather than directly comparing raw scores across heterogeneous metrics. The default threshold achieves higher FAA and CAA and lower FFM than both 0.01 and 0.10, supporting the selected drift sensitivity.

Reviewer_ySXtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies continual learning for vision-language models from a skill-evolution perspective. The paper proposes an approach SCALE which aims to retain reusable foundation skills like counting, grounding, and reasoning while continually learning on new tasks over time without replaying past downstream data. The method detects drifting foundation skills using held-out behavioral probes and skill-specific drift-aware gates. SCALE optimizes both the current-task and skill-maintenance objectives jointly. The method is evaluated on VQA-v3 using Molmo2-4B as the pre-trained VLM on eight sequential VQA tasks with eight foundation skills.

Critical strengths
  1. The proposed method is well explained with illustrations and details. The concept of separating a skill’s functionality while allowing its behavior to change is interesting.
  2. The analysis of behavioral evolution and skill functionality in section 4.4 is an useful analysis.
Critical weaknesses
  1. Poor ablations: The major components of the proposed method are not ablated, leading to poor novelty claims. Table 2 varies only the generation policy and Table 4 discusses the budget. The key design choices for the method proposed in section 3 are not ablated. This significantly reduces the significance of the formulations used to combine the components.

  2. The paper does not justify the selection of hyperparameters. How are these values selected?

  3. How do you position this work against compositional continual learning approaches [a,b] ? These works studies flexible composition of skills for continual learning. The paper lacks discussions and comparison to these domain of works.

[a] How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition. TMLR 2023.

[b] Separating the what and how of compositional computation to enable reuse and continual learning. NeurIPS 2025.

Questions for authors
  1. How does the proposed concept of skill-evolution extend to other domains like continual learning on image datasets?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ySXt [part 1]

We thank the reviewer for the constructive feedback and have revised the manuscript with additional experiments and clarifications.

[W1] Ablation of Key Components

To assess SCALE's core components, we conduct two ablations: w/o drift-aware gate (uniform allocation), which disables drift-aware skill selection and uniformly allocates the maintenance budget across all eight foundation skills, and w/o task-priority projection, which directly combines task and skill gradients without projection. All other training settings remain unchanged.

Table 1: Component ablation results on VQA-v3 under SS (%).

MethodLoc.Comm.TypeAct.ColorCountRecog.JudgeFAA↑CAA↑FFM↓
SCALE (Full)56.0592.0733.4668.9667.9463.2335.7890.7463.5362.462.91
w/o drift-aware gate (uniform allocation)57.1990.1130.3868.6537.0642.0235.6890.5556.4660.5010.43
w/o task-priority projection56.2092.0233.1668.6364.9163.2535.4090.5363.0161.222.86

As shown in Table 1, replacing drift-aware skill selection and allocation with uniform maintenance reduces FAA and CAA by 7.07 and 1.96 percentage points, respectively, while increasing FFM by 7.52 points. This highlights the importance of prioritizing foundation skills that exhibit functionality degradation. Removing task-priority projection slightly reduces FFM by 0.05 points but decreases FAA and CAA by 0.52 and 1.24 points, respectively. Thus, task-priority projection contributes to a more favorable stability–plasticity trade-off, rather than minimizing forgetting alone. See Table 4 of the revised manuscript for the corresponding component ablations.

[W2] Hyperparameter Selection and Sensitivity

To justify our hyperparameter choices, we conduct additional sensitivity analyses of the Point verification threshold, EMA smoothing coefficient (β\beta), and Gate activation threshold (τref\tau_{\mathrm{ref}}) on VQA-v3 under SS, and present the results in Table 2.

Table 2: Hyperparameter sensitivity on VQA-v3 under SS (%) (Default: Point verifier = 0.75, EMA (β=0.5\beta=0.5), Gate (τref=0.05\tau_{\mathrm{ref}}=0.05)).

SettingLoc.Comm.TypeAct.ColorCountRecog.JudgeFAA↑CAA↑FFM↓
SCALE (Full)56.0592.0733.4668.9667.9463.2335.7890.7463.5362.462.91
Point Verifier = 0.5055.7092.3629.7768.6024.1252.8535.0090.4756.1160.3210.76
Point Verifier = 0.9056.3792.5031.3068.5263.6154.0835.9590.5061.6062.033.71
EMA (β=0\beta=0)56.3888.2230.9068.5623.6053.9034.5890.4955.8359.358.47
EMA (β=0.9\beta=0.9)55.8587.7331.7770.4464.2561.9637.6390.7462.5561.392.70
Gate (τref=0.01\tau_{\mathrm{ref}}=0.01)56.8492.1630.4468.6360.9959.4235.4590.5761.8161.603.46
Gate (τref=0.10\tau_{\mathrm{ref}}=0.10)56.4487.7732.0468.5065.7756.1435.6990.6661.6361.134.50

i) Point Verification Threshold. Except for Point, the other seven foundation skills verify normalized predictions through exact matching against ground-truth annotations. Point instead predicts coordinate sets requiring tolerance for spatial deviations, so we use point-matching F1 with a separate verification threshold of 0.75. Compared with 0.50 and 0.90, the default improves FAA by 7.42 and 1.93 points, respectively, while achieving higher CAA and lower FFM.

ii) EMA Smoothing and Gate Activation. The default (β=0.5\beta=0.5) achieves higher FAA and CAA than both (β=0\beta=0) and (β=0.9\beta=0.9). Although (β=0.9\beta=0.9) slightly reduces FFM (2.70 vs. 2.91), it yields lower FAA and CAA. Similarly, the default (τref=0.05\tau_{\mathrm{ref}}=0.05) outperforms both 0.01 and 0.10 across FAA, CAA, and FFM.

Overall, the default configuration achieves the highest FAA and CAA among all tested hyperparameter settings, supporting our choices alongside the maintenance-budget sensitivity results in Table 6 of the revised manuscript. The corresponding hyperparameter results are provided in Table 5 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer ySXt [part 2]

[W3] Relationship to Compositional Continual Learning

We thank the reviewer for highlighting these relevant works. While compositional continual learning [a,b] investigates how reusable knowledge can be organized and composed across sequential tasks, SCALE differs in three fundamental aspects.

i) the source of reusable knowledge. Rather than learning or composing task-specific skill modules, SCALE exploits the generalizable foundation skills already acquired through large-scale VLM pretraining. Our focus is not on how skills are composed, but on how these pretrained capabilities can continually support downstream adaptation as the model evolves.

ii) the maintenance paradigm. SCALE decouples foundation-skill maintenance from downstream task history. Instead of preserving accumulated task knowledge, it monitors the functionality of pretrained foundation skills against a fixed pretrained reference and selectively maintains degraded skills through verifier-guided behavioral co-evolution, while prioritizing current-task adaptation.

iii) the resource scaling mechanism. SCALE maintains a fixed per-refresh budget, independent of the number and identities of previously learned tasks. This budget is dynamically allocated among foundation skills according to their measured degradation, rather than expanding maintenance resources as downstream tasks accumulate.

Together, these distinctions establish a different continual-learning perspective: sustaining the reusable capabilities inherited from pretraining, rather than composing or preserving accumulated task knowledge, with maintenance costs bounded independently of downstream task history.

References

[a] Mendez, J. A., and Eaton, E. How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition. TMLR, 2023.

[b] Shan, H., Sun, M., and Duncker, L. Separating the What and How of Compositional Computation to Enable Reuse and Continual Learning. NeurIPS, 2025.

[Q1] Generalization to Continual Learning on Image Datasets

We appreciate the reviewer's suggestion to investigate SCALE's applicability to continual learning on image datasets. To examine its cross-backbone generalization, we additionally evaluate SCALE on SmolVLM-Base, retaining the same framework, eight foundation-skill categories, and VQA-v3 SS continual-learning setting. Skill data are constructed from the model's publicly available training resources.

Table 3: Cross-backbone continual-learning results on VQA-v3 under SS (%).

MethodLoc.Comm.TypeAct.ColorCountRecog.JudgeFAA↑CAA↑FFM↓
SmolVLM SFT39.9373.7916.4259.508.180.7021.4575.5836.9441.4012.30
SmolVLM SCALE42.6176.2116.7160.278.861.6222.8476.5338.2143.769.27

As shown in Table 3, SCALE improves FAA and CAA by 1.27 and 2.36 percentage points, respectively, while reducing FFM by 3.03 points, with improvements across all eight downstream tasks. These results indicate that SCALE's benefits are not restricted to a single pretrained VLM backbone, suggesting its potential to support continual learning in other large pretrained models on downstream tasks involving image perception, recognition, and understanding, given appropriate foundation-skill supervision and verification mechanisms. See Table 2 of the revised manuscript for the corresponding cross-backbone comparison.

I thank the authors for the rebuttal responses. The experiments added during the rebuttal improves the paper.

However, several concerns are still not addressed.

The authors present a sensistivity analysis of the selected hyperparameters without answering "How are these values selected?" Are these values selected based on results on test set followng the provided analysis. This was my concern since the method involves several hyperparameters. The point is not to use the hyperparameters which gives the best results but how to select them without access to test set since validation sets in CL is tricky. Even in this analysis, the authors select values randomly with no explanation (0.1, 0.01 for threshold; 0.5, 0.9 for point verifier, 0, 0.9 for β\beta). I do not see any effort to discuss or resolve this.

The authors present a weak ablation table with ablations of only 2 components presented in the rebuttal while more components are still not properly ablated. what about the contributions from section 3.3 part?

I believe that the paper is not ready for publication. I maintain the same rating.

We thank the reviewer for the follow-up comments and would like to clarify the following two points.

Regarding hyperparameter selection, SCALE's default hyperparameters are primarily determined by the design principles and intended roles of its components, rather than solely by seeking the best performance under a particular experimental setting. Specifically, the Point verification threshold of 0.75 balances the quality of spatial matching against the availability of valid maintenance samples. The EMA coefficient of 0.5 balances responsiveness to functionality degradation with stability across maintenance refreshes. The Gate threshold of 0.05 filters minor functionality fluctuations and activates maintenance when meaningful degradation occurs relative to the pretrained reference. These hyperparameters remain fixed throughout continual adaptation without task-specific adjustments. The sensitivity analysis in Table 5 of the revised manuscript examines the effects of varying these values and is intended to assess the robustness of the default configuration rather than serve as the sole justification for parameter selection.

Regarding the ablations of Section 3.3, we would like to emphasize that the original submission already included policy-design ablations in Table 2 (Table 3 in the revised manuscript). These experiments compare SCALE with Direct PT Replay, Latest, and Stale under both SS and PS, examining the contributions of evolving behavioral supervision, the stage-entry generation policy, and refresh-level behavior regeneration. In addition, the revised manuscript includes independent ablations of drift-aware gating and task-priority projection in Table 4, as well as an analysis of Point verification threshold sensitivity in Table 5. Together, these experiments evaluate the major design choices underlying SCALE.

We appreciate the reviewer's continued feedback and hope these clarifications help address the concerns.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their constructive feedback and valuable suggestions. To address the major concerns raised by the reviewers, we have revised the manuscript and conducted additional experiments and analyses, as summarized below.

  1. Core-component ablations: We conduct independent ablations of drift-aware gating and task-priority projection to further validate SCALE's core mechanisms.
  2. Foundation-skill functionality evaluation: We correct the initial foundation-skill score, clarify the distinction between online monitoring and offline evaluation, and provide additional comparisons of foundation-skill functionality between SCALE and continual SFT.
  3. Foundation-skill activation and contributions: We report maintenance activation statistics for all eight foundation skills and conduct individual skill-removal ablations to examine their contributions to downstream continual-learning performance and knowledge retention.
  4. Task-aligned supervision and data-source dependence: We conduct additional experiments on task-aligned supervision and data-source dependence to further investigate the sources of SCALE's performance gains.
  5. Cross-backbone generalization: We extend SCALE to SmolVLM-Base to evaluate its applicability across different pretrained VLM backbones.
  6. Hyperparameter and budget sensitivity: We provide additional sensitivity analyses of the skill verification threshold, EMA coefficient, Gate activation threshold, and maintenance budget under PS.
  7. Methodological clarification and positioning: We further clarify behavioral co-evolution, skill-specific behavioral distance, fixed-budget maintenance, and the distinction between SCALE and compositional continual learning.

We have incorporated the corresponding experimental results and methodological clarifications into the revised manuscript. We sincerely appreciate the reviewers' valuable feedback, which has helped strengthen our experimental validation and more clearly articulate the contributions of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.