- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the multimodal safety issues when an open-weight VLM is specialized for professional domains. In the motivating study, safety tuning after domain training drops professional EM, and the reverse order drops the MSTS safe rate. Mixed-SFT preserves domain gains, but its safety varies a lot across seeds. This paper proposes the method CoLead for mitigating the problem. Firstly CoLead freezes the connector and run K learner-only SGD updates on professional data. And at the adapted state, normalizes the professional and safety gradients by fixed scales and pick alpha with MGDA. Finally, it jointly updates the connector and learner once, and carry the state into the next cycle. This proposed method achieves lower validation NLL compared to previous methods in professional domains.
Critical strengths
-
Problem setting and motivation is strong. Interference between domain specialization and safety is the frequent problem when the company release their own product.
-
Comparison is fair. The author use same initialization-validation subset-seed. And also they propose FLOPs matching curve for fairness (Fig. 4)
-
The authors use reasonable baseline Flattened-MGDA. This method use same MGDA, but it is better that the CoLead used separate inner professional update.
-
Coverage of domain and architecture is broad. The authors handled 3 domains and 2 architectures including post-hoc safety baselines like MMAligner and VLGuard-data SFT.
-
The authors disclose the limited honestly. The authors disclose the transfer failure on VQA-RAD, high hallucination rate, and some issues on monitor and hyperparameter search.
Critical weaknesses
-
The number of update confound. In Table 11, CoLead has 17 times for learner update and 1 time for joint update. But Mixed-SFT and Flattened-MGDA have only 1 time for joint update. The author matches the number of samples and FLOPs but on constant-LR SGD there is 18 times differences. And also there is difference on joint learner LR (on finance domain 0.225 vs 0.125). So there is not alternating SFT baseline with "small batch and domain step many times + safety step", I cannot decide the gains come from the method or just the number of steps.
-
It seems hierarchical formulation doesn't really show up in the method. The essence of the claimed coupling is that the response changes as φ changes. Because CoLead doesn't differentiate through the trajectory, this term never enters the connector gradient. After the joint update, θ also leaves the argmin constraint, which is never re-enforced. So the method is alternating optimization ("K steps of domain SFT, then an MGDA joint step"), and the bilevel framing is mostly explanatory. The authors themselves concede in the Remark that there is no convergence guarantee.
-
MGDA contributes essentially nothing. In Fig. 5, fixed α (0.75, 0.825, 0.9) produces nearly identical trajectories to MGDA. With α clipped to [0.75, 0.9], MGDA likely behaves almost like a fixed weight. That leaves the schedule as the only contribution, which makes weakness 1 even more damaging.
-
Over-refusal is never measured. Only unsafe rates are reported, with no XSTest- or MOSSBench-style metric. In fact, the MSSBench sample in F.1.2 has the id :safe, meaning a benign context, yet CoLead's refusal to give a distance is framed positively as "calibration." It should be checked whether finance safety NLL dropping to Safety-SFT levels reflects a drift toward refusal.
-
The motivation study is disconnected from the main experiments. CoLead is never run in the Figure 1 remote-sensing setting. That study also uses a very short budget (25 steps, 400 samples), which may exaggerate the size of the interference.
-
There is no mechanistic evidence for connector coupling. The motivation draws on the representation-shift literature (Lee et al. 2025; Liu et al. 2025), but there is no analysis of how the connector actually changes visual representations or alignment with a refusal direction. The only support is a 25-cycle short ablation where freezing the connector raises safety NLL from 1.44 to 1.64.
Questions for authors
Firstly, please refer the Critical Weaknesses section. I reconstruct the questions using above weaknesses.
-
CoLead performs 17 learner updates plus 1 joint update per cycle, while Mixed-SFT and Flattened-MGDA perform a single joint update (Table 11). Could you report an alternating-SFT baseline with matched update counts and learning rates? For example, K=17 domain-only steps (batch 16) followed by one Mixed-SFT or fixed-α joint step per cycle. If CoLead's advantage largely disappears, what remains of the claimed contribution beyond the schedule?
-
In finance, CoLead uses a joint learner LR of 0.225, while Mixed-SFT and Flattened-MGDA use 0.125 (Table 10). How were these chosen, and how do the baselines perform at 0.225 (or CoLead at 0.125)? Given that hyperparameter search budgets were not matched, can you report the number of configurations tried per method?
-
The formulation's motivation is that θ*(φ) depends on φ, but CoLead's connector gradient omits dθ*/dφ. In what precise sense does CoLead exploit connector–learner coupling, beyond evaluating gradients at a post-adaptation state? Have you tried even a cheap approximation of the implicit term (e.g., one-step unrolled or finite-difference) to see whether it matters empirically?
-
Fig. 5 shows fixed α ∈ {0.75, 0.825, 0.9} behaves almost identically to MGDA. What is the empirical distribution of α^(t) during training, and how often does it hit the interval bounds? If it is mostly clipped, should the method be described as fixed-weight coordination?
-
Safety results report only unsafe-assistance rates. Can you add an over-refusal evaluation, such as XSTest, MOSSBench, or the safe split of MSSBench reported separately? In particular, the F.1.2 example (mssbench:...:safe) involves a benign context, and CoLead declines to give a distance. How often does this happen across the safe split?