- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces EvoSteer, a method for steering the refusal behavior of an LLM-based agent (in particular, memory-based ones that retrieves memories and incorporates them into its context). Prior work has shown that such memory augmentation can reduce refusal rates compared with memory-free baselines. EvoSteer first estimates a refusal-related direction at each layer using a fit set. At inference time, it requires two forward passes, one with memory and one without. It projects the difference between their representations at selected token positions onto , then counteracts the execution-ward component during the memory-augmented forward pass. The results suggest that EvoSteer achieves a better balance between safety-related refusal and task performance. EvoSteer is evaluated on Mind2Web-SC and SafeAgentBench, covering two memory methods and four base LLMs across the experiments. It shows promising recovery of refusal behavior on Qwen3-14B and Qwen3-32B.
Critical strengths
- The paper addresses under-refusal in memory-based agents, a pressing concern given their utility and increasing adoption.
- The motivation for EvoSteer is clear, and the methodology is reasonable. The paper is well organized, and the technical details are clearly explained. The analysis in Figure 2 provides useful support for the method’s design.
Critical weaknesses
-
The ablation study in Section 4.4 is not convincing. EvoSteer has a relatively small effect on Qwen3.6-27B, and the AWM memory baseline does not reduce the aggregate refusal rate compared with the memory-free baseline. This makes the setting less informative for evaluating recovery from memory-induced under-refusal. Moreover, several differences in Figure 3 are only 2.5 percentage points, corresponding to a single task, and may reflect evaluation variability. These results provide limited support for the conclusions drawn from the ablations.
-
Important baselines are missing:
- As discussed around line 122, a fixed steering vector can be applied to adjust the model’s refusal boundary. In this setting, a natural baseline would directly steer along the same estimated from the fit set, with a fixed steering strength tuned on that set. This comparison would help establish the benefit of EvoSteer’s adaptive correction over simpler steering, particularly given the additional reference forward pass it requires.
- Another possible baseline is a lightweight probing classifier trained on representations from the fit set to predict whether a request should be refused. Would such an approach be applicable to this setting, and how would it compare with EvoSteer?
Questions for authors
See weaknesses above.