- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents a multidirectional method to suppress refusal. Stage 1 selects which experts to use. Stage 2 selects a multidirectional prototype to steer the refusal suppression. The method aims to improve refusal suppression while preserving general model capabilities. The authors report an average HarmBench-judged target compliance rate of 96.33%, outperforming the strongest baseline in each model–benchmark setting by an average of 12.72 percentage points.
Critical strengths
S1. Consistent gains with minimal intervention. LoMC achieves the highest TCR across all four models and four benchmarks while largely preserving general capability accuracy (GCA). Notably, it does so by editing only a small subset of experts rather than making broad changes to the model's weights. This consistency across settings, combined with the targeted nature of the edit, is the main reason I view the contribution as meaningful.
Critical weaknesses
-
W1. It’s not clear that multiple prototypes are actually needed. Table 2 tests K = 2, 4, and 8, plus a support-only variant that is effectively K = 0, but there’s no K = 1. Since the prototypes all get merged into one rank-one direction per layer anyway, I’d like to know whether the jump from 69.17% to 89.17% TCR really comes from combining several prototypes, or whether adding just one prototype to the base direction would get you most of the way there.
-
W2. The main tables report no standard deviations or confidence intervals. The selection of E = 6, K = 4 rests on it having the best TCR and average GCA among tested configurations, but its advantage over K = 8 is small, and no significance testing shows the difference is meaningful. This makes it difficult to judge whether the chosen configuration, and the reported margins over baselines, are robust.
-
W3. The compliance–capability trade-off is not fully addressed. LoMC consistently achieves the highest TCR, but not always the highest GCA. The paper does not discuss how much additional refusal suppression justifies a capability loss, particularly in settings where competing methods preserve more capability. A clearer trade-off analysis (e.g., a Pareto comparison) would help establish whether LoMC is strictly preferable.
Questions for authors
Q1. The choice of baselines is not justified/clear, why choose these methods to compare to?
3 comments
Thank you
Thank you for the additional results. The responses address most of my concerns.
The new K=1 ablation provides convincing evidence that combining multiple prototypes meaningfully improves TCR. I encourage the authors to incorporate both the Pareto analysis and the baseline selection rationale into the revised paper.
Response to Reviewer Xqdi
Thank you for the positive assessment. We have incorporated both points into the revised manuscript. The baseline selection rationale is now stated in Section 5.1 and elaborated in Appendix A.6. The Pareto analysis is summarized in Section 5.5, with the complete comparison across all 16 settings provided in Appendix E.2, Table 26. These additions are highlighted in blue.
We sincerely thank you for your thoughtful and constructive feedback, which has helped us improve the clarity, experimental analysis, and overall quality of the paper.
Response to Reviewer Xqdi
Dear Reviewer,
Thank you for your time and consideration. We would greatly appreciate it if you could take a moment to review our rebuttal and share any further feedback. We would be happy to address any remaining concerns.
Thank you again for your valuable time!