- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper, the authors study the goal drift problem in LLM agents and multi-agent systems. Goal drift refers to the pattern in which systems tend to deviate from the overall goal as execution unfolds in long-horizon tool-use and inter-agent collaboration systems. To address this problem, the authors propose using a steering agent that supervises execution on the fly and provides signals to steer the system back when goal drift happens. In their method, they jointly train the two paths with GRPO with a pairwise ranking objective. The authors provide experiments on information-seeking search tasks to show the effectiveness of their method. Based on their experiments, they claim that the learned steering model generalizes to OOD tasks and can transfer to unseen executors.
Critical strengths
(1) Good writing: The paper is well written and easy to read.
(2) Experimental validation on both single-agent and multi-agent systems: Although the experiments are still limited in some aspects (see the Weaknesses and Questions sections), the experiments in both single-agent and multi-agent domains show the method’s generalizability.
Critical weaknesses
I will consider raising my rating if my concerns and questions are well addressed.
(1) The motivation for the RL-learned steering system can be further improved. See my Question 4. Providing supporting experiments on this can resolve this concern.
(2) Limited experiments: baselines in the multi-agent experiments. In Table 4, the baselines used in single-agent experiments, such as planning and verification, are missing. The discussion is also limited: the paper can be improved with more analysis of why RL on the steering model is the best combination among the tested baselines. For example, in many tasks, SFT on correct RL traces (which was termed filtered BC in the paper) can achieve performance similar to that of RL checkpoints. Why does this method make a significant performance difference in learning steering models? Is it because it cannot reduce bad behaviors? More discussion on this can give more insight. In addition, selecting stronger steering models, for example, the Qwen3.5 series, in the experiments can better show the generalizability of the proposed method.
Questions for authors
(1) How much performance gain is obtained from the steering model (Qwen3-8B) in Table 1? Can you provide an oracle steering model, for example, a frontier closed-source model, and use it in the experiments as a reference?
(2) In the generalization experiments shown in Table 3, there is a performance gap between the steering model and the executor model: Qwen2.5-7B is weaker than Qwen3-8B, and all the tested models perform more poorly than Qwen2.5-7B when there is no intervention. I am curious about the performance of stronger models: for example, using Qwen3.5-4B/9B as the executor and seeing how it generalizes to the other end of the range of executor models.
(3) Since you utilize the Qwen3-8B model for intervention, you should also report the performance of the Qwen3-8B model as the executor model (with and without another Qwen3-8B model as the steering model).
(4) Can you compare the RL-learned intervention system with a single-model RL-learned executor model? For example, with the same base model, Qwen3-8B, use this model as the single model, perform GRPO on it on the training dataset, and keep its training resources (GPU hours) the same. After training, report the performance of this executor-only system. This question can help us better understand the advantages and disadvantages of adding steering models to the system.
(5) The objectives of the intervention system are to maximize task utility rewards while reducing the intervention cost. Could you discuss this work in a broader context, such as multi-objective optimization in LLM post-training? An example is reducing user intervention burdens while maximizing task performance [1, 2, 3].
References:
[1] CollabLLM: From passive responders to active collaborators. ICML 2025
[2] Training Proactive and Personalized LLM Agents. COLM 2026
[3] Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization. ICML 2026
4 comments
Response to Reviewer xZCA (2/2)
Q5: Discussion in the broader context of multi-objective optimization in LLM post-training.
A: We thank the reviewer for pointing out these related works. These works [1, 2, 3] train proactive agents that elicit the user's needs under ambiguous instructions, e.g., by asking clarifying questions, and balance task performance against the user's interaction effort. In contrast, we build a steering agent that monitors and intervenes on agent and multi-agent systems in place of the user, and its cost is controlled by the steering frequency at deployment (Fig. 4). We would also like to clarify that our objective is to maximize the task utility by steering an existing agent system with an external steering agent, rather than to trade the task utility against the intervention cost. On top of this, we let the fast path learn to prioritize high-impact states, so that the task utility is improved as much as possible under a fixed steering frequency. The two objectives therefore do not conflict and lie on two orthogonal dimensions, which differs from conventional multi-objective optimization. Note that the trade between steering cost and task performance in Section 4.3 is made at inference by adjusting the steering frequency, which is separate from multi-objective optimization during training. We will include this discussion in the revision.
W2: Planning and verification baselines in Table 4, why RL outperforms filtered BC, and stronger steering models.
A:
Planning and verification baselines in Table 4
In response, we supplement planning and verification baselines in the multi-agent setting:
| Method | Avg. | Δ |
|---|---|---|
| No Intervention | 42.49 | – |
| Planning | 41.65 | −0.84 |
| Planning w/ RL | 42.98 | +0.49 |
| Verification | 38.82 | −3.67 |
| Verification w/ RL | 44.47 | +1.98 |
| Steering | 35.34 | −7.14 |
| Steering w/ RL (ours) | 47.83 | +5.34 |
The trend is consistent with Table 1, where RL improves all three supervision forms and steering achieves the largest gain. We will add these baselines to Table 4 in the revision.
Why RL outperforms filtered BC
Filtered BC is an offline algorithm that imitates a fixed set of positive steering actions, so its training data does not track the improving policy and it converges quickly. In contrast, RL in our paper is on-policy and keeps improving on the actions sampled from its latest policy. This is consistent with prior findings, where online training is comparable to offline training on filtered samples in the early stage and gains a clear advantage in the later stage [4], and RL outperforms filtered BC in training device-control agents [5].
To further verify this, we evaluate the intermediate checkpoints of the two methods:
| Training progress | 0% | 20% | 50% | 100% |
|---|---|---|---|---|
| w/ Filtered BC | 44.5 | 45.2 | 45.6 | 44.1 |
| w/ RL (ours) | 44.5 | 46.5 | 48.1 | 49.4 |
From the results, filtered BC improves in the early stage of training (44.5 → 45.6), while its performance drops in the later stage (44.1) due to overfitting to the training set. In contrast, RL keeps learning from the data sampled by the on-policy model and maintains the improvement throughout training (44.5 → 49.4). We will add this analysis in the revision.
Stronger steering models
As suggested, we further adopt the stronger Qwen3.5-9B as the steering backbone and report its zero-shot steering result:
| Steering backbone | Zero-Shot Δ |
|---|---|
| Qwen3-8B (ours) | +1.84 |
| Qwen3.5-9B | +2.67 |
The results show that our framework can use a stronger steering backbone to bring a larger zero-shot gain (+2.67 vs. +1.84).
Due to the high training cost, we are not able to finish a new training run with Qwen3.5-9B as backbone in a short time. We will update the results once we obtain them and add them to the revision.
References
[4] Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.
[5] Bai et al. DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning. NeurIPS, 2024.
Kind Reminder and Additional Results
Dear Reviewer xZCA,
Thank you again for your detailed review. As the discussion period is closing soon, we would like to kindly check whether our responses have addressed your concerns.
Regarding W2, we now provide the RL result with a stronger steering backbone. We train Qwen3.5-9B as the steering backbone with the same RL algorithm to further verify the generalizability of our method. This experiment is only partially completed, and we attach the latest intermediate result below:
| Steering backbone | Zero-Shot Δ | RL Δ |
|---|---|---|
| Qwen3-8B (ours) | +1.84 | +6.66 |
| Qwen3.5-9B | +2.67 | +7.92 (60 steps) |
With only 60 training steps, the Qwen3.5-9B steering agent already reaches a gain of +7.92, higher than the Qwen3-8B agent trained for 300 steps (+6.66). This shows that a stronger backbone gives a stronger steering agent in both the zero-shot and RL settings, and further supports that our RL algorithm transfers to other backbones.
We would be glad to clarify any remaining concerns, and would appreciate it if you could consider re-evaluating our work in light of these results.
Sincerely,
The Authors
Follow-up Before the Discussion Deadline
Dear Reviewer xZCA,
We understand this is a very busy period, and we apologize for following up again. As the discussion period closes in less than one day, we would be grateful for your feedback on whether our responses have addressed your concerns. We would be glad to clarify any remaining questions before the discussion closes.
Best regards,
The Authors
Response to Reviewer xZCA (1/2)
Q1: The gain obtained from the steering backbone, and a frontier model as an oracle steering agent.
A: In Table 1, the steering model brings a gain of 6.66 points over No Intervention, of which the untrained Qwen3-8B backbone contributes 1.84 points (Zero-Shot Steering) and RL contributes the remaining 4.82 points. As suggested, we further use frontier models as the steering agent for reference, including the open-weight Qwen3.8-max and the closed-source GPT-6.1-sol, under the same steering frequency as Table 1.
Our steering agent with the 8B backbone reaches 69% of the gain of Qwen3.8-max (+6.66 vs. +9.60) and exceeds GPT-6.1-sol (+4.98).
Q2: Generalization to stronger executors such as Qwen3.5-4B and Qwen3.5-9B.
A: As suggested, we evaluate the same RL steering agent without retraining on executors with Qwen3.5-4B and Qwen3.5-9B as backbones, following the setting of Table 3:
Both Qwen3.5 executors are stronger than Qwen2.5-7B without intervention (58.47 and 56.43 vs. 42.70). RL steering still improves them by +1.89 and +3.69 without retraining. The gain is smaller than on Qwen2.5-7B (+6.66), since stronger executors are less prone to goal drift on the search tasks we tested.
However, we expect the steering paradigm to bring more benefit to strong executors when tasks get harder, such as GAIA. To verify this, we evaluate the Qwen3.5-4B executor on GAIA, where the steering agent improves the performance from 27.18 to 32.52, an improvement of around 20%. Due to the limited API budget, we leave the exploration of more complex tasks to future work.
Q3: Qwen3-8B as the executor, with and without a Qwen3-8B steering agent.
A: We select Qwen2.5-7B as the executor because we observe clear goal drift on it. We now extend our experiment and report the results with Qwen3-8B as the executor below, where the steering agent uses the same Qwen3-8B backbone:
The results are consistent with our expectation. The untrained Qwen3-8B can hardly improve the Qwen3-8B executor as the steering agent (+0.81). In contrast, the steering agent trained with the Qwen2.5-7B executor demonstrates its transferability and brings a gain of +3.94.
W1 & Q4: Comparison with a single-model executor trained by RL.
A: As suggested, we keep the training GPU hours the same and train the executor directly with GRPO. We keep the executor backbone setting the same as that of the steering agent, i.e., Qwen3-8B (Reasoning). Note that the GRPO-trained executor reasons at every step, which is much more costly at inference than our default setting in Table 1, so the two are not directly comparable. We therefore compare it with our system under a 100% steering frequency (the rightmost data point in Fig. 4), where a reasoning model is also involved at every step:
The results show that the two settings achieve comparable performance (59.8 vs. 59.6). However, the steering framework additionally provides the ability to switch the executor without retraining (Table 3) and the ability to flexibly adjust the inference cost at deployment (Fig. 4).