acceptodds
Withdrawn from ICLR 2027

Connector–Learner Coupling for Safe Domain Specialization of Vision–Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Specializing vision–language models (VLMs) for professional domains requires improving domain competence while preserving multimodal safety. Balancing these objectives is challenging: domain adaptation can weaken safety behavior, while safety tuning can degrade performance on legitimate tasks. We study this interaction through Connector–Learner Coupling, which captures how the visual connector shapes the representations used for language-side professional learning and how the resulting learner state, in turn, shapes professional and safety gradients for connector updates. This perspective leads to a hierarchical formulation in which professional adaptation defines a connector-conditioned learner response, while professional and safety objectives are evaluated on the same coupled model state. Based on this formulation, we develop CoLead, a first-order co-adaptation method. In each cycle, CoLead first adapts the learner through multiple professional updates with the connector fixed, then jointly updates both components using normalized professional and safety gradients evaluated at the adapted state. The updated connector–learner pair carries forward across cycles without differentiating through the adaptation trajectory. Across finance, radiology, and autonomous driving, CoLead achieves lower mean validation negative log-likelihood on both professional and safety data than mixed fine-tuning and flattened gradient coordination under matched data exposure. Generation-based evaluations further show competitive professional performance alongside improved safety-related measures on multiple benchmarks, while revealing limitations in hallucination control and cross-dataset transfer.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-language models, multimodal safety, domain specialization, connector–learner coupling, hierarchical optimization, multi-objective learning, parameter-efficient fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_RkcAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the multimodal safety issues when an open-weight VLM is specialized for professional domains. In the motivating study, safety tuning after domain training drops professional EM, and the reverse order drops the MSTS safe rate. Mixed-SFT preserves domain gains, but its safety varies a lot across seeds. This paper proposes the method CoLead for mitigating the problem. Firstly CoLead freezes the connector and run K learner-only SGD updates on professional data. And at the adapted state, normalizes the professional and safety gradients by fixed scales and pick alpha with MGDA. Finally, it jointly updates the connector and learner once, and carry the state into the next cycle. This proposed method achieves lower validation NLL compared to previous methods in professional domains.

Critical strengths
  • Problem setting and motivation is strong. Interference between domain specialization and safety is the frequent problem when the company release their own product.

  • Comparison is fair. The author use same initialization-validation subset-seed. And also they propose FLOPs matching curve for fairness (Fig. 4)

  • The authors use reasonable baseline Flattened-MGDA. This method use same MGDA, but it is better that the CoLead used separate inner professional update.

  • Coverage of domain and architecture is broad. The authors handled 3 domains and 2 architectures including post-hoc safety baselines like MMAligner and VLGuard-data SFT.

  • The authors disclose the limited honestly. The authors disclose the transfer failure on VQA-RAD, high hallucination rate, and some issues on monitor and hyperparameter search.

Critical weaknesses
  • The number of update confound. In Table 11, CoLead has 17 times for learner update and 1 time for joint update. But Mixed-SFT and Flattened-MGDA have only 1 time for joint update. The author matches the number of samples and FLOPs but on constant-LR SGD there is 18 times differences. And also there is difference on joint learner LR (on finance domain 0.225 vs 0.125). So there is not alternating SFT baseline with "small batch and domain step many times + safety step", I cannot decide the gains come from the method or just the number of steps.

  • It seems hierarchical formulation doesn't really show up in the method. The essence of the claimed coupling is that the response changes as φ changes. Because CoLead doesn't differentiate through the trajectory, this term never enters the connector gradient. After the joint update, θ also leaves the argmin constraint, which is never re-enforced. So the method is alternating optimization ("K steps of domain SFT, then an MGDA joint step"), and the bilevel framing is mostly explanatory. The authors themselves concede in the Remark that there is no convergence guarantee.

  • MGDA contributes essentially nothing. In Fig. 5, fixed α (0.75, 0.825, 0.9) produces nearly identical trajectories to MGDA. With α clipped to [0.75, 0.9], MGDA likely behaves almost like a fixed weight. That leaves the schedule as the only contribution, which makes weakness 1 even more damaging.

  • Over-refusal is never measured. Only unsafe rates are reported, with no XSTest- or MOSSBench-style metric. In fact, the MSSBench sample in F.1.2 has the id :safe, meaning a benign context, yet CoLead's refusal to give a distance is framed positively as "calibration." It should be checked whether finance safety NLL dropping to Safety-SFT levels reflects a drift toward refusal.

  • The motivation study is disconnected from the main experiments. CoLead is never run in the Figure 1 remote-sensing setting. That study also uses a very short budget (25 steps, 400 samples), which may exaggerate the size of the interference.

  • There is no mechanistic evidence for connector coupling. The motivation draws on the representation-shift literature (Lee et al. 2025; Liu et al. 2025), but there is no analysis of how the connector actually changes visual representations or alignment with a refusal direction. The only support is a 25-cycle short ablation where freezing the connector raises safety NLL from 1.44 to 1.64.

Questions for authors

Firstly, please refer the Critical Weaknesses section. I reconstruct the questions using above weaknesses.

  • CoLead performs 17 learner updates plus 1 joint update per cycle, while Mixed-SFT and Flattened-MGDA perform a single joint update (Table 11). Could you report an alternating-SFT baseline with matched update counts and learning rates? For example, K=17 domain-only steps (batch 16) followed by one Mixed-SFT or fixed-α joint step per cycle. If CoLead's advantage largely disappears, what remains of the claimed contribution beyond the schedule?

  • In finance, CoLead uses a joint learner LR of 0.225, while Mixed-SFT and Flattened-MGDA use 0.125 (Table 10). How were these chosen, and how do the baselines perform at 0.225 (or CoLead at 0.125)? Given that hyperparameter search budgets were not matched, can you report the number of configurations tried per method?

  • The formulation's motivation is that θ*(φ) depends on φ, but CoLead's connector gradient omits dθ*/dφ. In what precise sense does CoLead exploit connector–learner coupling, beyond evaluating gradients at a post-adaptation state? Have you tried even a cheap approximation of the implicit term (e.g., one-step unrolled or finite-difference) to see whether it matters empirically?

  • Fig. 5 shows fixed α ∈ {0.75, 0.825, 0.9} behaves almost identically to MGDA. What is the empirical distribution of α^(t) during training, and how often does it hit the interval bounds? If it is mostly clipped, should the method be described as fixed-weight coordination?

  • Safety results report only unsafe-assistance rates. Can you add an over-refusal evaluation, such as XSTest, MOSSBench, or the safe split of MSSBench reported separately? In particular, the F.1.2 example (mssbench:...:safe) involves a benign context, and CoLead declines to give a distance. How often does this happen across the safe split?

Reviewer_Kt5cOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes COLEAD for safe domain specialization of vision–language models. Each cycle alternates between professional updates to language-side LoRA parameters and a joint connector–learner update using MGDA to coordinate professional and safety objectives. Experiments span finance, radiology, and autonomous driving. While the problem is relevant, concerns about methodological novelty, experimental confounds, and the strength of the safety evidence limit the contribution.

Critical strengths
  • The motivation is clear. Preserving safety while improving domain competence is an important problem for specialized VLMs.
  • The method is sample. The method trains only the connector and language-side LoRA parameters while freezing the remaining backbone, avoiding differentiation through the adaptation trajectory.
  • The evaluation seems broad. Experiments cover multiple domains and model architectures, providing evidence beyond a single task or backbone.
Critical weaknesses
  1. One critical weakness is that the method lacks novelty.
    The hierarchical formulation requires an optimal professional learner response to the connector. However, the algorithm uses fixed-step adaptation without assessing response approximation error, computing response-dependent hypergradients, or ensuring that joint updates preserve the constraint. The paper does not adequately establish how this procedure solves or approximates the stated problem. Consequently, the contribution appears primarily to combine existing LoRA fine-tuning, alternating updates, and MGDA, with insufficient evidence that the proposed structure provides a substantive methodological advance.

  2. The experiment lacks a fair baseline comparison and cannot support the main claim.
    COLEAD performs 17 professional updates and one joint update per cycle, whereas Mixed-SFT and Flattened-MGDA accumulate the corresponding examples into a single joint update. This changes update counts, effective batch sizes, and optimization trajectories; some learning rates also differ, and hyperparameter-search budgets are not matched. Matching sample exposure or approximate FLOPs does not eliminate these confounds. Stronger baselines controlling optimization schedules are needed to determine whether the gains arise from the proposed coupling mechanism rather than more effective professional optimization.

  3. The evidence for safe specialization remains limited.
    COLEAD’s medical hallucination rate is 89.61%, exceeding both the base model’s 81.72% and Mixed-SFT’s 75.94%. It also underperforms Domain-SFT on VQA-RAD and Mixed-SFT on MM-AU. These results show that reduced harmful assistance does not consistently translate into improved professional reliability. Moreover, evaluation relies primarily on a single LLM judge, lacks independent medical expert validation, and uses text-only judging for MM-AU. The evidence supports improvements in selected benchmark trade-offs, but does not establish broadly reliable safe specialization.

  4. Presentation could be clearer.
    Figure 2 is visually crowded, and its overlapping conceptual elements make the relationship between the hierarchical formulation and the training procedure difficult to follow. A simpler diagram with clearer dependencies would improve readability.

Questions for authors

In my view, the paper’s central idea is essentially the joint fine-tuning of LoRA parameters and the VLM’s connector. The method is overly simple and lacks sufficient novelty. I consider this a fundamental weakness that a rebuttal is unlikely to resolve.

Reviewer_WMShOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies safe domain specialization of vision–language models (VLMs): fine-tuning an open-weight VLM for a professional domain (finance, radiology, autonomous driving) while preserving multimodal safety. A controlled remote-sensing study motivates the problem by showing interference — safety tuning after domain tuning cuts professional performance by ~21pp, the reverse order cuts safety by ~19pp, and Mixed-SFT is high-variance. The authors propose a Connector–Learner Coupling view: the visual connector φ shapes the representations fed to the language model, and the language-side learner θ (adapted with LoRA) learns professional competence on top of them, so the two should be coordinated rather than treated independently. This is formalized hierarchically, with professional adaptation defining a connector-conditioned learner response and both objectives evaluated at the resulting adapted pair. The method, COLEAD, is a first-order co-adaptation loop: each cycle adapts the learner with the connector fixed, then computes professional and safety gradients at the adapted state, combines them via MGDA (with the safety weight constrained to [α_min, α_max]), and jointly updates both blocks without differentiating through the adaptation trajectory. Across three domains with Qwen2-VL (2B/7B) and LLaVA-1.5, COLEAD attains lower mean validation NLL on both professional and safety data than Mixed-SFT and Flattened-MGDA, with generation-based performance close to Domain-SFT at reduced unsafe-response rates, while the authors explicitly report limitations in hallucination control and cross-dataset transfer.

Critical strengths
  • The problem is clearly framed and the central view is well-motivated and internally consistent. Decomposing safe specialization into a connector that shapes representations and a learner that acquires professional competence, and then coordinating safety against competence at the professionally adapted state rather than the current state, is a clean and well-reasoned idea that gives a concrete reason to prefer COLEAD over coordinating at the un-adapted state (its Flattened-MGDA baseline). The mechanism is supported by the paper's own ablations: freezing the learner raises both professional and safety NLL, whereas freezing the connector mainly raises safety NLL, which is direct evidence that the connector carries safety-relevant representations and that the two blocks should be adapted together.

  • The empirical study is carefully controlled. COLEAD is evaluated across three distinct professional domains with three-seed means and standard deviations, matched data exposure across the compared methods, and a matched-FLOPs comparison of the learning curves (Figure 4) that rules out the differences being merely a larger training-compute budget, alongside component, objective, and sample-allocation ablations that trace how each design choice affects competence and safety. The paper is also candid about where COLEAD does not win — higher hallucination on ROCOv2, weaker cross-dataset transfer, and a non-best VQA-RAD score — which makes its positive claims easier to trust.

Critical weaknesses
  • Gains on the generation-based metrics are often small and fall within seed variance, with no significance testing. For example, in finance the FinVis score is 5.174±0.256 for COLEAD versus 5.173±0.180 for Domain-SFT and 5.031±0.173 for Mixed-SFT, and MME-Finance is 1.788 versus 1.789 for Domain-SFT — effectively tied with heavily overlapping error bars. Against Flattened-MGDA, the baseline that most directly isolates the proposed contribution, COLEAD's means are consistently better but the professional-score standard deviations overlap substantially. The clean, separable advantage is largely on validation NLL, which is a weak proxy for actual professional correctness and real safety, so the evidence on the deployment-relevant generation metrics is thinner than the headline suggests, and no paired significance test is reported.

  • Regressions on hallucination and transfer weaken the overall "safe specialization" claim. On ROCOv2 the hallucination rate for COLEAD (89.61%) is higher than the base model (81.72%), i.e., specialization worsens factuality, and on driving MM-AU Mixed-SFT (3.667) exceeds COLEAD (3.509). For medical and driving use, hallucination is itself a core safety property, so the claim of simultaneously improving safety holds only on some measures. The authors disclose this honestly, but it does limit the strength of the central conclusion.

  • The formal apparatus is heavier than what the method delivers, and the generation metrics rely on a single LLM judge. The hierarchical, set-valued response formulation S_prof(φ) is elaborate, but COLEAD is a first-order heuristic approximation (finite inner updates, no differentiation through the trajectory), and the authors state that MGDA's guarantees "do not imply convergence of the full COLEAD recurrence," so the formalism is largely motivational while the actual contribution is an engineering training loop. Separately, all generation-based professional and safety scores come from a single LLM judge (gpt-5.6-sol) and hallucination is flagged automatically, with no human or expert (especially radiologist) verification, leaving the key behavioral metrics dependent on one evaluator.

Questions for authors
  1. On the generation-based metrics, the differences between COLEAD and Mixed-SFT / Flattened-MGDA often lie within seed variance. Can paired across-seed significance tests be provided, and is the clearly separable advantage essentially confined to validation NLL?

  2. On ROCOv2 the hallucination rate exceeds the base model (89.61% vs. 81.72%). How is this reconciled with the central "safe specialization" claim — does connector–learner coordination trade factuality for other safety measures, and is there a direct comparison targeting hallucination?

  3. The professional and safety scores come from a single judge (gpt-5.6-sol) and hallucination from automated flags. Can an independent judge or a small expert check (particularly for radiology) be added to assess judge dependence and the robustness of the reported metrics?

  4. The key thing that sets COLEAD apart from Flattened-MGDA is that it adapts the learner first and then balances safety against competence at that adapted state. Intuitively, why does balancing at the adapted state work better than balancing at the current state — what goes wrong in the latter, and is there a concrete example or simple diagnostic that shows this difference in action?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.