- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes RESPOND, a continual on-policy distillation framework that consolidates independently trained LoRA task experts into a single student. It uses forgetting-risk estimates to select historical experts and LoRA response probes to identify informative current-task trajectories for distillation. Experiments across three MLLM backbones report improved accuracy and retention on an eight-task stream.
Critical strengths
-
Jointly selecting historical experts according to forgetting risk and trajectory-level response addresses a meaningful challenge in continual OPD.
-
Experiments across three backbones, component ablations, and proxy-correlation analyses provide encouraging evidence for the proposed selection mechanisms.
Critical weaknesses
-
It remains unclear whether supervision on current-task trajectories adequately protects historical capabilities. The selected probes may have limited coverage, especially for short responses or mismatched tasks, while stage-wise risk resets may overlook previously degraded capabilities.
-
The practical CL advantage is not fully established. Independent expert training, a growing expert library, and repeated probing introduce unreported costs. The custom evaluation stream and large performance degradation of some baselines also warrant further validation.
Questions for authors
-
Since historical experts are distilled only on current-task student trajectories, what evidence shows that risk- and response-guided selection effectively preserves historical capabilities, beyond capturing high teacher KL signals? How does the method behave when current and historical tasks have little overlap?
-
How are capabilities already degraded before a new stage protected, given that the risk reference and persistent states are reset?
-
What is the total computational cost of RESPOND, including expert training, probing, and consolidation, compared with compute-matched CL baselines? As the number of tasks grows, how do expert storage, probing, and selection costs scale, despite the limit of three historical-teacher queries per update?
-
Could the authors explain the substantial performance differences of SDPO across backbones, particularly on COCO (25.22 with Qwen3.5-4B versus 74.89 with Qwen3-VL-4B), and clarify whether the baseline configurations were appropriately tuned for each model?