acceptodds
Under review as a conference paper at ICLR 2027

LoMC: Localized Multidirectional Correction for Refusal Suppression in Routed Foundation Models

Abstract

We study whether localized post-training weight edits can suppress refusal in safety-aligned routed models without substantially degrading general capabilities. Sparse routing offers localized editing sites, but single-direction estimation may miss diverse refusal patterns, whereas broad edits risk degrading general capabilities. We introduce Localized Multidirectional Correction (LoMC), which separates edit-support selection where to edit from correction-direction estimation (how to correct). Stage I ranks expert output projections by weight alignment with the harmful–benign mean activation difference, then uses validation under capability and edit-budget constraints to fix a sparse set of expert projection matrices as the routed edit support. This limits the scope of routed weight changes to reduce the risk of capability degradation. With this support fixed, Stage II restarts from the original checkpoint and constructs harmful-prompt activation prototypes, each defining a candidate direction relative to the benign activation mean. It selects four prototype-derived directions, favoring alignment with the harmful–benign mean-difference direction while discouraging redundancy, then aggregates them with this base direction. The aggregation incorporates diverse activation patterns into one correction direction per edited layer, with each selected matrix receiving an update of rank at most one. This design combines richer direction estimation with localized weight changes to strengthen refusal suppression while limiting capability degradation. Across four routed vision-language models and four safety benchmarks, LoMC achieves 96.33% mean HarmBench-judged target compliance, exceeding the best baseline in each setting by over 12.72% on average while maintaining near-original average scores on four general capability benchmarks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Refusal Suppression, Routed Foundation Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XqdiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a multidirectional method to suppress refusal. Stage 1 selects which experts to use. Stage 2 selects a multidirectional prototype to steer the refusal suppression. The method aims to improve refusal suppression while preserving general model capabilities. The authors report an average HarmBench-judged target compliance rate of 96.33%, outperforming the strongest baseline in each model–benchmark setting by an average of 12.72 percentage points.

Critical strengths

S1. Consistent gains with minimal intervention. LoMC achieves the highest TCR across all four models and four benchmarks while largely preserving general capability accuracy (GCA). Notably, it does so by editing only a small subset of experts rather than making broad changes to the model's weights. This consistency across settings, combined with the targeted nature of the edit, is the main reason I view the contribution as meaningful.

Critical weaknesses
  • W1. It’s not clear that multiple prototypes are actually needed. Table 2 tests K = 2, 4, and 8, plus a support-only variant that is effectively K = 0, but there’s no K = 1. Since the prototypes all get merged into one rank-one direction per layer anyway, I’d like to know whether the jump from 69.17% to 89.17% TCR really comes from combining several prototypes, or whether adding just one prototype to the base direction would get you most of the way there.

  • W2. The main tables report no standard deviations or confidence intervals. The selection of E = 6, K = 4 rests on it having the best TCR and average GCA among tested configurations, but its advantage over K = 8 is small, and no significance testing shows the difference is meaningful. This makes it difficult to judge whether the chosen configuration, and the reported margins over baselines, are robust.

  • W3. The compliance–capability trade-off is not fully addressed. LoMC consistently achieves the highest TCR, but not always the highest GCA. The paper does not discuss how much additional refusal suppression justifies a capability loss, particularly in settings where competing methods preserve more capability. A clearer trade-off analysis (e.g., a Pareto comparison) would help establish whether LoMC is strictly preferable.

Questions for authors

Q1. The choice of baselines is not justified/clear, why choose these methods to compare to?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Xqdi

Dear Reviewer,

Thank you for your time and consideration. We would greatly appreciate it if you could take a moment to review our rebuttal and share any further feedback. We would be happy to address any remaining concerns.

Thank you again for your valuable time!

Reviewer_XqdiOpenReview ↗

Thank you

Thank you for the additional results. The responses address most of my concerns.

The new K=1 ablation provides convincing evidence that combining multiple prototypes meaningfully improves TCR. I encourage the authors to incorporate both the Pareto analysis and the baseline selection rationale into the revised paper.

AuthorsOpenReview ↗

Response to Reviewer Xqdi

Thank you for the positive assessment. We have incorporated both points into the revised manuscript. The baseline selection rationale is now stated in Section 5.1 and elaborated in Appendix A.6. The Pareto analysis is summarized in Section 5.5, with the complete comparison across all 16 settings provided in Appendix E.2, Table 26. These additions are highlighted in blue.

We sincerely thank you for your thoughtful and constructive feedback, which has helped us improve the clarity, experimental analysis, and overall quality of the paper.

Reviewer_bwCQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Existing refusal edits do not adequately resolve the trade-off between suppressing refusal and preserving general capabilities in routed vision-language models. The paper proposes LoMC, which selects a sparse edit support and aggregates prototype-derived directions for localized weight updates. Experiments on text-only and multimodal safety benchmarks report stronger harmful-response compliance than the compared methods while retaining similar average performance on general capability evaluations.

Critical strengths
  1. Separating support selection from direction estimation provides a clear way to study localized refusal editing.

  2. Evaluation across multiple routed backbones and input modalities shows substantial behavioral effects, with capability measurements reported alongside attack success.

Critical weaknesses
  1. The method mainly combines existing expert selection and prototype-based refusal editing. It remains unclear what new insight is gained beyond combining these components into a localized weight update.

  2. The selection score measures weight alignment, while the controls do not match routing frequency or update magnitude. Because the mechanism study covers only one model and benchmark, it does not firmly establish why the selection rule improves the trade-off.

  3. Capability retention is assessed through an average over a small set of tasks, without uncertainty estimates. This makes it difficult to judge whether small differences between methods reflect reliable preservation of general capabilities.

Questions for authors

How does LoMC compare with routing-frequency-based expert selection when tensor coverage, update norm, and the prototype estimator are matched?

1 comment
AuthorsOpenReview ↗

Response to Reviewer bwCQ

Dear Reviewer,

Thank you for your time and consideration. We would greatly appreciate it if you could take a moment to review our rebuttal and share any further feedback. We would be happy to address any remaining concerns.

Thank you again for your valuable time!

Reviewer_f6VfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This work studies post-training weight edits that suppress refusal in routed MoE vision-language models while trying to keep general capabilities intact. It first selects a sparse set of expert projections as the edit support, then builds a multi-prototype correction direction and applies a rank-one update only inside that support. Across four models and four safety benchmarks, the method reaches high HarmBench target-compliance rates with capability scores close to the original checkpoints. This work studies post-training weight edits that suppress refusal in routed MoE vision-language models while trying to keep general capabilities intact. It first selects a sparse set of expert projections as the edit support, then builds a multi-prototype correction direction and applies a rank-one update only inside that support. Across four models and four safety benchmarks, the method reaches high HarmBench target-compliance rates with capability scores close to the original checkpoints.

Critical strengths
  • The direction of work is interesting: looking at localized refusal suppression inside the sparse experts of routed models.
  • The two-stage method is coherent and reasonably easy to follow once the notation is clear.
  • Figure 4’s zoom into the high-GCA region makes the TCR-GCA trade-off easy to read.
  • The ablations on the number of selected experts EE and prototype directions KK (Table 2) are useful for seeing that the trade-off is not monotonic.
  • The evaluation carefully separates non-refusal from HarmBench-judged target compliance and keeps validation and test prompts separate.
Critical weaknesses
  • The abstract sentence "then uses validation under capability and edit-budget constraints to fix a sparse set of expert projection matrices as the routed edit support" is too condensed and hard to parse. An abstract should be easier to follow.
  • Figure 1 is too packed and difficult to read.
  • Figure 2 is also quite dense. It mixes routes, schematic plots, and other elements; the behavioral-effect / target-compliance sketches in particular are hard to interpret.
  • The main ablations for EE and KK are reported only on one backbone and one benchmark.
  • The limitations are only discussed in the appendix, and the text there is again too difficult to parse for a human reader.
  • In the "Refusal representations and safety interventions" paragraph of the related work, it would be useful to also mention Agnihotri et al. (2025) as a closely related study that looks at how well models can judge their own refusal behavior under abliteration: Agnihotri, S., Jakubassa, J., Dey, P., Goyal, S., Schiele, B., Radhakrishnan, V. B., & Keuper, M. (2025). A granular study of safety pretraining under model abliteration. In NeurIPS Workshop Lock-LLM: Prevent Unauthorized Knowledge Use from Large Language Models. https://openreview.net/forum?id=dUHRkgPMej
Questions for authors
  1. Why were E=6E=6 and K=4K=4 fixed as the main setting rather than chosen per model or per benchmark on the validation set?
  2. Could the abstract and Figures 1 & 2 be simplified so the core "where to edit / how to correct" idea is clearer without the dense schematics?
  3. How sensitive are the reported TCR numbers to the HarmBench judge, and were any of the high-compliance generations manually inspected?
  4. Would it be possible to briefly discuss the limitations in the main paper as well, while referring to a more detailed (and hopefully easier to read) limitations section in the appendix?
1 comment
AuthorsOpenReview ↗

Response to Reviewer f6Vf

Dear Reviewer,

Thank you for your time and consideration. We would greatly appreciate it if you could take a moment to review our rebuttal and share any further feedback. We would be happy to address any remaining concerns.

Thank you again for your valuable time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.