- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper targets instruction-hierarchy violations, where a model follows a user turn or a tool result over the system prompt. The authors note that the conflicting low-priority content is usually also task-relevant, so the method must resolve the conflict and preserve context at the same time. RoCAST is a training-free inference-time method whose distinguishing part is offline head selection. From role-swapped calibration variants, it locates the first-divergence token and measures single-head intervention effects, then computes three per-head statistics: a role-swapped benefit score, a role-response consistency sign from a leave-one-head-out consensus, and a preservation score from the no-conflict system-side margin. Heads are ranked by a standardized combination of the benefit and preservation scores, the top K=96 are steered at inference by a per-role multiplicative bias on attention logits with system set to +1 and tool set to -1 and beta=4, and the reweighting preserves each head's total attention mass. Calibration uses 64 instruction pairs and 250 paragraphs. Across five models the conflict metric improves by 21.3 to 33.1 points.
Critical strengths
- The stage-wise ablation in Table 4 is unusually informative: benefit-only selection on Qwen3-8B drops aligned behavior by 18.4 points, while adding the consistency filter gains 12.2 on conflict and 13.4 on aligned simultaneously, which is direct evidence that single-head benefits are not additive.
- The random-head control applies the same operator at the same strength and budget (8.5 versus 26.3 average points), which correctly separates the selection criterion from the intervention.
Critical weaknesses
- The abstract's claim of keeping aligned task execution within 1.3 points of native holds only for that one domain.
- The negative tool-role bias is not supported by ablation study.
Questions for authors
- What are the per-domain preservation bounds, and would the abstract claim be restated as domain-specific?
- What happens with the tool-role bias set to 0, holding K, beta and the normalization fixed?
- Is K=96 tuned per model or transferred, and how sensitive is the conflict metric over K from 32 to 192?
3 comments
Response to Reviewer #2
Q.3 Head-Budget Selection and Sensitivity
As detailed in Appendix A.2 and Table 6, we selected independently for each model using the same predefined stopping criterion on 800 held-out Beyond Oracle pairs. For all five models, the improvement from to was significant, whereas the next step from to was not. Thus, the stopping rule independently selected for every model, rather than transferring a budget chosen on one model to the others. We kept , , and the head ranking fixed throughout selection, without using any IHEval examples.
Head-budget sensitivity is presented in Figure 3 and Appendix C (Figure 6 and Table 10) on a separate 760-prompt IHEval subset. We evaluated , subject to each model's eligible-set size. Although conflict accuracy changes nonmonotonically as increases, the gains over Native persist across all evaluated feasible budgets. Overall, these results demonstrate that RoCAST's conflict improvements are robust across a range of head budgets, rather than depending on the specific choice of .
We clarified the preservation claim in the abstract and add the tool-role bias ablation to the appendix. We hope these clarifications and additional results address your concerns.
Gentle Follow-up on Author Response
Thank you again for your careful review and helpful feedback.
As the discussion period draws to a close, we wanted to briefly follow up on our rebuttal. We believe that our clarifications and additional experiments have addressed the main concerns raised in your review.
We would appreciate hearing whether these responses have sufficiently resolved your questions, or if there is anything further we could clarify.
Thank you again for your time and consideration.
Response to Reviewer #1
Thank you for the encouraging assessment of our ablation studies. We address your three questions below, including a new ablation of the negative tool-role bias.
Q.1 Domain-Specific Preservation
As reported in Table 1, the preservation claim applies specifically to IHEval's task-execution domain. Across the five models, the average aligned accuracy and its changes relative to Native are:
Despite these domain-specific regressions, RoCAST preserves aligned performance substantially better than IHDec and V-Simple. Since IHDec excludes tool-use tasks, we report both the comparable non-tool average and the overall IHEval average across five models:
RoCAST retains near-native aligned accuracy on the non-tool evaluation, with substantially smaller losses than IHDec and V-Simple. Beyond IHEval, RoCAST improves IFEval-sys by 5.6 points over Native on average and incurs smaller MATH-500 losses than both methods. Unlike these baselines, RoCAST also preserves Native MMLU accuracy by remaining inactive when no system or tool messages are present.
We will clarify the domain-specific scope in the abstract:
Q.2 Ablation of the Negative Tool-Role Bias
To assess the contribution of the negative tool-role bias, we removed only this term while keeping the selected heads, K=96, β=4, and normalization fixed. The results are provided in Common Table A.
Removing the term improves aligned tool-use accuracy by 8.1 points on Qwen3-8B and 12.3 points on Qwen3-14B, but reduces tool-conflict accuracy by 8.5 and 13.7 points, respectively. On Llama-3.1-8B, matched reruns show a 0.6-point aligned improvement with no change in conflict accuracy.
The ablation confirms that the negative tool-role bias plays an important role in tool-conflict resolution on Qwen3. Removing it partially recovers aligned accuracy but substantially reduces conflict accuracy, demonstrating that the term helps prevent models from following conflicting instructions in tool outputs. Moreover, the remaining aligned degradation indicates that tool suppression is not the sole source of preservation loss. These results highlight the importance of the negative tool-role bias in RoCAST, as it improves resistance to conflicting tool instructions.