- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Dynamic Role Activation Steering (DRAS) to reduce cases where an agent performs another role's tasks. It extracts a role vector by comparing activations with and without role instructions, and trains a linear detector on compliant and overstepping trajectories. During generation, the detector controls how strongly the role vector is applied. Experiments with ChatDev and MetaGPT using two backbone models show fewer role violations, higher software Quality scores, and lower runtime and token use.
Critical strengths
-
DRAS modifies activations at one model layer without updating the backbone weights or adding a response-regeneration step. Table 3 reports substantially lower offline memory and computation costs than fine-tuning a separate LoRA model for each role.
-
Quality improves and both runtime and token use fall relative to Vanilla MAS in all four system–model combinations. The authors use separate datasets for offline preparation and final evaluation, and check a sample of the LLM judge's decisions against human annotations.
Critical weaknesses
-
DRAS requires clearly defined role responsibilities and does not cover collaboration without a clear division of work. All experiments and baseline comparisons are also limited to software development. The method has not been evaluated in other domains.
-
In Table 4, static steering produces fewer role violations, runs faster, and uses fewer tokens, while DRAS achieves higher Quality scores. Both approaches have advantages, but the paper does not explain how the fixed strength was chosen or how changing it affects quality and efficiency. Without this comparison, it is hard to tell when to prefer one approach over the other.
-
The detector is trained to judge whether a completed trajectory contains overstepping, but it is used to assess risk while generation is still in progress. The paper does not explain why this training should work for real-time detection. More evaluation is needed beyond the single example in Figure 5.
Questions for authors
-
How was the static steering strength selected and tuned in Table 4? How do Quality, runtime, and token use change with different fixed strengths?
-
What is the source of the 2.5B Qwen2.5 model shown in Figure 1b (page 2) and mentioned in the accompanying text (lines 69–70)? The official Qwen2.5 model cards do not list this variant.
6 comments
Author's Response (2)
[W3] The detector is trained to judge whether a completed trajectory contains overstepping, but it is used to assess risk while generation is still in progress. The paper does not explain why this training should work for real-time detection. More evaluation is needed beyond the single example in Figure 5.
Response: Thank you for the question. The approach of training probes on activations from complete responses and using them for real-time detection and dynamic control of steering strength during generation has also been adopted in ACT [1] and AAC [2]. Other studies on internal representations [3,4] also identified predictive signals in internal model states before the corresponding errors or violations appear in the output. These studies provide some support for the feasibility of this approach.
To further evaluate real-time role-overstepping detection beyond the single example in Figure 5, we kept the detector unchanged and conducted an additional prefix evaluation for the CEO role in ChatDev using Qwen2.5-7B. We selected 100 trajectories with role overstepping and 100 trajectories without role overstepping, all generated without steering. For trajectories with overstepping, we annotated the onset of the first role-overstepping behavior and recorded the probe’s role-overstepping predicted probability (p) at 20, 10, 5, and 1 generation steps before this onset. For trajectories without overstepping, we used the final generated token as the reference position and recorded the probabilities at the same offsets. The table reports the mean and standard deviation of predicted probability at each position for each group.
| 20 steps before | 10 steps before | 5 steps before | 1 step before | |
|---|---|---|---|---|
| With overstepping: onset of the first violation | 0.36 ± 0.105 | 0.54 ± 0.092 | 0.79 ± 0.095 | 0.83± 0.075 |
| Without overstepping: final generated token | 0.29± 0.100 | 0.35± 0.124 | 0.42 ± 0.101 | 0.31± 0.103 |
The results show that, for trajectories that subsequently overstep, the probe’s predicted probability rises noticeably before the first violation occurs. DRAS uses precisely this signal to increase steering strength before overstepping occurs, thereby reinforcing role adherence. For trajectories without overstepping, the predicted probability remains low at the evaluated positions. These results support the detector's ability to detect early warning signals of role overstepping before it occurs.
[1] Tianlong Wang et al. Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categories. WWW, 2025.
[2] Eric Yocam et al. Adaptive Activation Cancellation for Hallucination Mitigation in Large Language Models. 2026.
[3] Ben Snyder, Marius Moisescu, and Muhammad Bilal Zafar. On Early Detection of Hallucinations in Factual Question Answering. KDD, 2024.
[4] Dhananjay Ashok and Jonathan May. Language Models Can Predict Their Own Behavior. NeurIPS, 2025.
[Q2] What is the source of the 2.5B Qwen2.5 model shown in Figure 1b (page 2) and mentioned in the accompanying text (lines 69–70)? The official Qwen2.5 model cards do not list this variant.
Response: Thank you for pointing this out. We checked our experimental code and confirmed that the model used was Qwen2.5-3B. The "2.5B" label in Figure 1b was a labeling error introduced when preparing the plot, where "2.5" in the model family name was mistakenly used as the model size. The correct model sizes are 14B, 7B, and 3B, and we will correct this error in both Figure 1b and the corresponding text.
Thank you for the responses and experiments. However, my concern about applicability remains, as the effectiveness of DRAS beyond software development has not been evaluated.
Besides, for the comparison with static steering, does DRAS still offer a better trade-off between task quality and efficiency when the static steering strength is tuned independently?
Author's Response (1)
Thank you for providing your valuable and constructive feedback! Please find below the responses to each comment on weaknesses and questions.
Response: Thank you for this comment. This work focuses on role-based multi-agent collaboration. We evaluate DRAS on software development tasks because software development is a widely used and well-established application setting for such multi-agent systems [1-3]. However, we would like to clarify that the design of DRAS is based on role definitions and internal model activations and is not specific to software development. Therefore, the same procedure can be applied to other collaborative tasks with clearly defined roles for generation-time role control. We will add this discussion to the revised manuscript.
[1] Yue Hu et al. Self-Evolving Multi-Agent Collaboration Networks for Software Development. ICLR, 2025.
[2] Zhengren Wang et al. MaintainCoder: Maintainable Code Generation Under Dynamic Requirements. NeurIPS, 2025.
[3] Iris Xu et al. BOAD: Discovering Hierarchical Software Engineering Agents via Bandit Optimization. ICLR, 2026.
Response: Thank you for the question. (i) To isolate the effects of different steering strategies, Static Steering and Binary Gating use the same base steering strengths as DRAS. (ii) For Static Steering, weak steering may be insufficient to correct overstepping, while sufficiently strong steering may disrupt role-adherent generation and produce garbled outputs, leading to premature termination and task failure. Therefore, Static Steering’s lower token consumption in the reported results is partly due to the premature termination of some failed tasks, rather than more efficient completion of successful tasks.