acceptodds
Under review as a conference paper at ICLR 2027

Learning to Steer LLM-based Agent Systems

Abstract

Large language model (LLM)-based agents and multi-agent systems increasingly tackle complex tasks through long-horizon tool use and inter-agent collaboration. However, these systems are prone to deviating from the overall goal as execution unfolds, which wastes execution resources and leads to undesired outcomes, a failure termed goal drift. In this work, we propose a steering agent that supervises execution on the fly and steers the system back on track when drift occurs. The steering agent observes the execution context, deciding when to intervene with a low-cost fast path and how to intervene with a high-cost slow path. To unlock its steering intelligence, the ability to assess goal drift and generate effective interventions, we jointly train the two paths with reinforcement learning, combining a pairwise ranking objective with group-relative policy optimization (GRPO). Experiments on information-seeking search tasks show that the steering agent provides effective supervision for both single-agent and multi-agent systems, improving their performance and efficiency. The learned steering intelligence generalizes to out-of-domain tasks, transfers to unseen executors, and supports controllable intervention frequency without retraining.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agent, Goal Drift, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xZCAOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study the goal drift problem in LLM agents and multi-agent systems. Goal drift refers to the pattern in which systems tend to deviate from the overall goal as execution unfolds in long-horizon tool-use and inter-agent collaboration systems. To address this problem, the authors propose using a steering agent that supervises execution on the fly and provides signals to steer the system back when goal drift happens. In their method, they jointly train the two paths with GRPO with a pairwise ranking objective. The authors provide experiments on information-seeking search tasks to show the effectiveness of their method. Based on their experiments, they claim that the learned steering model generalizes to OOD tasks and can transfer to unseen executors.

Critical strengths

(1) Good writing: The paper is well written and easy to read.

(2) Experimental validation on both single-agent and multi-agent systems: Although the experiments are still limited in some aspects (see the Weaknesses and Questions sections), the experiments in both single-agent and multi-agent domains show the method’s generalizability.

Critical weaknesses

I will consider raising my rating if my concerns and questions are well addressed.

(1) The motivation for the RL-learned steering system can be further improved. See my Question 4. Providing supporting experiments on this can resolve this concern.

(2) Limited experiments: baselines in the multi-agent experiments. In Table 4, the baselines used in single-agent experiments, such as planning and verification, are missing. The discussion is also limited: the paper can be improved with more analysis of why RL on the steering model is the best combination among the tested baselines. For example, in many tasks, SFT on correct RL traces (which was termed filtered BC in the paper) can achieve performance similar to that of RL checkpoints. Why does this method make a significant performance difference in learning steering models? Is it because it cannot reduce bad behaviors? More discussion on this can give more insight. In addition, selecting stronger steering models, for example, the Qwen3.5 series, in the experiments can better show the generalizability of the proposed method.

Questions for authors

(1) How much performance gain is obtained from the steering model (Qwen3-8B) in Table 1? Can you provide an oracle steering model, for example, a frontier closed-source model, and use it in the experiments as a reference?

(2) In the generalization experiments shown in Table 3, there is a performance gap between the steering model and the executor model: Qwen2.5-7B is weaker than Qwen3-8B, and all the tested models perform more poorly than Qwen2.5-7B when there is no intervention. I am curious about the performance of stronger models: for example, using Qwen3.5-4B/9B as the executor and seeing how it generalizes to the other end of the range of executor models.

(3) Since you utilize the Qwen3-8B model for intervention, you should also report the performance of the Qwen3-8B model as the executor model (with and without another Qwen3-8B model as the steering model).

(4) Can you compare the RL-learned intervention system with a single-model RL-learned executor model? For example, with the same base model, Qwen3-8B, use this model as the single model, perform GRPO on it on the training dataset, and keep its training resources (GPU hours) the same. After training, report the performance of this executor-only system. This question can help us better understand the advantages and disadvantages of adding steering models to the system.

(5) The objectives of the intervention system are to maximize task utility rewards while reducing the intervention cost. Could you discuss this work in a broader context, such as multi-objective optimization in LLM post-training? An example is reducing user intervention burdens while maximizing task performance [1, 2, 3].

References:

[1] CollabLLM: From passive responders to active collaborators. ICML 2025

[2] Training Proactive and Personalized LLM Agents. COLM 2026

[3] Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization. ICML 2026

4 comments
AuthorsOpenReview ↗

Response to Reviewer xZCA (1/2)

Q1: The gain obtained from the steering backbone, and a frontier model as an oracle steering agent.

A: In Table 1, the steering model brings a gain of 6.66 points over No Intervention, of which the untrained Qwen3-8B backbone contributes 1.84 points (Zero-Shot Steering) and RL contributes the remaining 4.82 points. As suggested, we further use frontier models as the steering agent for reference, including the open-weight Qwen3.8-max and the closed-source GPT-6.1-sol, under the same steering frequency as Table 1.

Steering agentΔ
Qwen3.8-max+9.60
GPT-6.1-sol+4.98
RL Steering (ours, Qwen3-8B)+6.66

Our steering agent with the 8B backbone reaches 69% of the gain of Qwen3.8-max (+6.66 vs. +9.60) and exceeds GPT-6.1-sol (+4.98).


Q2: Generalization to stronger executors such as Qwen3.5-4B and Qwen3.5-9B.

A: As suggested, we evaluate the same RL steering agent without retraining on executors with Qwen3.5-4B and Qwen3.5-9B as backbones, following the setting of Table 3:

MethodQwen2.5-7B Avg.ΔQwen3.5-4B Avg.ΔQwen3.5-9B Avg.Δ
No Intervention42.70–58.47–56.43–
Zero-Shot Steering44.54+1.8459.05+0.5858.75+2.32
RL Steering (Ours)49.36+6.6660.36+1.8960.12+3.69

Both Qwen3.5 executors are stronger than Qwen2.5-7B without intervention (58.47 and 56.43 vs. 42.70). RL steering still improves them by +1.89 and +3.69 without retraining. The gain is smaller than on Qwen2.5-7B (+6.66), since stronger executors are less prone to goal drift on the search tasks we tested.

However, we expect the steering paradigm to bring more benefit to strong executors when tasks get harder, such as GAIA. To verify this, we evaluate the Qwen3.5-4B executor on GAIA, where the steering agent improves the performance from 27.18 to 32.52, an improvement of around 20%. Due to the limited API budget, we leave the exploration of more complex tasks to future work.


Q3: Qwen3-8B as the executor, with and without a Qwen3-8B steering agent.

A: We select Qwen2.5-7B as the executor because we observe clear goal drift on it. We now extend our experiment and report the results with Qwen3-8B as the executor below, where the steering agent uses the same Qwen3-8B backbone:

MethodAvg.Δ
No Intervention51.60–
Zero-Shot Steering52.41+0.81
RL Steering55.54+3.94

The results are consistent with our expectation. The untrained Qwen3-8B can hardly improve the Qwen3-8B executor as the steering agent (+0.81). In contrast, the steering agent trained with the Qwen2.5-7B executor demonstrates its transferability and brings a gain of +3.94.


W1 & Q4: Comparison with a single-model executor trained by RL.

A: As suggested, we keep the training GPU hours the same and train the executor directly with GRPO. We keep the executor backbone setting the same as that of the steering agent, i.e., Qwen3-8B (Reasoning). Note that the GRPO-trained executor reasons at every step, which is much more costly at inference than our default setting in Table 1, so the two are not directly comparable. We therefore compare it with our system under a 100% steering frequency (the rightmost data point in Fig. 4), where a reasoning model is also involved at every step:

MethodAvg.Training GPU h
Qwen2.5-7B w/ Steering59.8545
Qwen3-8B w/ GRPO59.6544

The results show that the two settings achieve comparable performance (59.8 vs. 59.6). However, the steering framework additionally provides the ability to switch the executor without retraining (Table 3) and the ability to flexibly adjust the inference cost at deployment (Fig. 4).

AuthorsOpenReview ↗

Response to Reviewer xZCA (2/2)

Q5: Discussion in the broader context of multi-objective optimization in LLM post-training.

A: We thank the reviewer for pointing out these related works. These works [1, 2, 3] train proactive agents that elicit the user's needs under ambiguous instructions, e.g., by asking clarifying questions, and balance task performance against the user's interaction effort. In contrast, we build a steering agent that monitors and intervenes on agent and multi-agent systems in place of the user, and its cost is controlled by the steering frequency at deployment (Fig. 4). We would also like to clarify that our objective is to maximize the task utility by steering an existing agent system with an external steering agent, rather than to trade the task utility against the intervention cost. On top of this, we let the fast path learn to prioritize high-impact states, so that the task utility is improved as much as possible under a fixed steering frequency. The two objectives therefore do not conflict and lie on two orthogonal dimensions, which differs from conventional multi-objective optimization. Note that the trade between steering cost and task performance in Section 4.3 is made at inference by adjusting the steering frequency, which is separate from multi-objective optimization during training. We will include this discussion in the revision.


W2: Planning and verification baselines in Table 4, why RL outperforms filtered BC, and stronger steering models.

A:

Planning and verification baselines in Table 4

In response, we supplement planning and verification baselines in the multi-agent setting:

MethodAvg.Δ
No Intervention42.49–
Planning41.65−0.84
Planning w/ RL42.98+0.49
Verification38.82−3.67
Verification w/ RL44.47+1.98
Steering35.34−7.14
Steering w/ RL (ours)47.83+5.34

The trend is consistent with Table 1, where RL improves all three supervision forms and steering achieves the largest gain. We will add these baselines to Table 4 in the revision.

Why RL outperforms filtered BC

Filtered BC is an offline algorithm that imitates a fixed set of positive steering actions, so its training data does not track the improving policy and it converges quickly. In contrast, RL in our paper is on-policy and keeps improving on the actions sampled from its latest policy. This is consistent with prior findings, where online training is comparable to offline training on filtered samples in the early stage and gains a clear advantage in the later stage [4], and RL outperforms filtered BC in training device-control agents [5].

To further verify this, we evaluate the intermediate checkpoints of the two methods:

Training progress0%20%50%100%
w/ Filtered BC44.545.245.644.1
w/ RL (ours)44.546.548.149.4

From the results, filtered BC improves in the early stage of training (44.5 → 45.6), while its performance drops in the later stage (44.1) due to overfitting to the training set. In contrast, RL keeps learning from the data sampled by the on-policy model and maintains the improvement throughout training (44.5 → 49.4). We will add this analysis in the revision.

Stronger steering models

As suggested, we further adopt the stronger Qwen3.5-9B as the steering backbone and report its zero-shot steering result:

Steering backboneZero-Shot Δ
Qwen3-8B (ours)+1.84
Qwen3.5-9B+2.67

The results show that our framework can use a stronger steering backbone to bring a larger zero-shot gain (+2.67 vs. +1.84).

Due to the high training cost, we are not able to finish a new training run with Qwen3.5-9B as backbone in a short time. We will update the results once we obtain them and add them to the revision.


References

[4] Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.

[5] Bai et al. DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning. NeurIPS, 2024.

AuthorsOpenReview ↗

Kind Reminder and Additional Results

Dear Reviewer xZCA,

Thank you again for your detailed review. As the discussion period is closing soon, we would like to kindly check whether our responses have addressed your concerns.

Regarding W2, we now provide the RL result with a stronger steering backbone. We train Qwen3.5-9B as the steering backbone with the same RL algorithm to further verify the generalizability of our method. This experiment is only partially completed, and we attach the latest intermediate result below:

Steering backboneZero-Shot ΔRL Δ
Qwen3-8B (ours)+1.84+6.66
Qwen3.5-9B+2.67+7.92 (60 steps)

With only 60 training steps, the Qwen3.5-9B steering agent already reaches a gain of +7.92, higher than the Qwen3-8B agent trained for 300 steps (+6.66). This shows that a stronger backbone gives a stronger steering agent in both the zero-shot and RL settings, and further supports that our RL algorithm transfers to other backbones.

We would be glad to clarify any remaining concerns, and would appreciate it if you could consider re-evaluating our work in light of these results.

Sincerely,

The Authors

AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline

Dear Reviewer xZCA,

We understand this is a very busy period, and we apologize for following up again. As the discussion period closes in less than one day, we would be grateful for your feedback on whether our responses have addressed your concerns. We would be glad to clarify any remaining questions before the discussion closes.

Best regards,

The Authors

Reviewer_Ms6LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problem of goal drift in LLM-based agents executing long-horizon tasks. The authors propose an external steering agent that observes the task goal and execution history during execution, uses the fast path to estimate whether the current state is worth intervening on, and uses the slow path to generate concrete intervention actions such as note and retry. The authors also freeze the task executor and train only the steering agent with reinforcement learning. Experiments are conducted primarily on information-seeking QA tasks, as well as out-of-domain search tasks, unseen executors, and a five-role multi-agent system. Overall, the results support the effectiveness of the proposed approach in improving task performance.

Critical strengths
  1. The paper proposes a practically meaningful online supervision framework. Compared with planning only before execution or verification only after execution, runtime steering can intervene before errors accumulate, making it suitable for long-horizon tool use and inter-agent handoffs.

  2. The fast-path and slow-path design is cost-conscious. The fast path performs low-cost screening of critical states, while the slow path generates detailed interventions only when necessary. At deployment time, adjusting the threshold provides a way to trade off task performance against intervention cost.

Critical weaknesses
  1. The experimental scope is insufficient to support strong claims about generality. The in-domain, out-of-domain, and executor-transfer experiments are still primarily conducted in search-based QA settings. The paper does not evaluate other commonly used domains, such as coding or everyday work tasks.

  2. The necessity of the fast-path/slow-path design is not sufficiently validated. The paper does not quantify how much performance is sacrificed, if any, in exchange for how much cost reduction compared with using the slow path throughout. It also does not report the accuracy of the fast-path intervention judgment itself.

Questions for authors
  1. Is the proposed method equally effective in other domains, such as coding or everyday work tasks? More specifically, can a steering model trained in the web-search domain generalize to non-search domains?

  2. What is the trade-off between the accuracy and cost of the fast path? For example, compared with using slow-path reasoning at every observation where the agent performs full reasoning before deciding whether and how to intervene, rather than simply answering yes or no, how much cost does the current fast-path/slow-path design save, and how does its task performance compare?

  3. The paper uses the stronger Qwen3-8B as the backbone of the steering agent. How does the capacity or capability of the backbone model affect the performance of the steering agent?

4 comments
AuthorsOpenReview ↗

Response to Reviewer Ms6L

W1 & Q1: Generality beyond search, and whether a steering agent trained on search generalizes to coding.

A: We further validate our steering agent design on coding tasks. We follow DeepCoder [1] and evaluate on LiveCodeBench-v5 and Codeforces, with mini-swe-agent [2] as the executor. Using the steering agent trained only on search tasks, we observe a consistent improvement in the average pass rate over the two benchmarks:

ExecutorMethodAvg.Δ
Qwen2.5-7BNo Intervention9.35–
w/ Steering Agent11.98+2.63
Qwen3-8BNo Intervention18.64–
w/ Steering Agent22.41+3.77

The improvement holds for both Qwen2.5-7B and Qwen3-8B as the executor (+2.63 and +3.77), which shows that the steering agent remains effective beyond search tasks. We will add the coding results in the revision.


W2 & Q2: Necessity of the fast path, its cost and performance against using the slow path throughout, and the accuracy of its judgment.

A: We thank the reviewer for suggesting this ablation. In response, we supplement an analysis of how the fast path affects performance and cost. As suggested, we design a variant that removes the fast path and lets the slow path of our trained steering agent reason at every state to decide whether and how to intervene.

DesignAvg.ΔSteering tokens per task (input / output)Latency
Slow path only52.04+9.3429.4k / 6.5k111 s
Fast path + slow path (ours)49.36+6.6620.6k / 1.7k34 s

Without the fast path, the gain increases from +6.66 to +9.34, while the average latency per task increases from 34 s to 111 s (3.3×). The fast path therefore saves 69% of the latency at the cost of 2.68 points, and its threshold allows users to choose this trade-off at deployment without retraining (Fig. 4).

We further measure how RL improves the judgment accuracy of the fast path. We compare the fast path before and after training on whether its steering priority ranks the states in line with their actual impact on the outcome, and report the AUROC metric [3], which measures the probability that a high-impact state receives a higher priority than a low-impact one. RL raises the AUROC from 0.54 to 0.63, which helps the fast path prioritize the states where steering matters (Fig. 4). We will add both results in the revision.


Q3: How the capacity of the backbone affects the steering agent.

A: We replace the steering backbone and report the gain over the no intervention baseline on the seven in-domain benchmarks with the Qwen2.5-7B executor:

Steering backboneZero-Shot Δ
Qwen3-4B+0.45
Qwen3-8B (ours)+1.84
Qwen3.5-9B+2.67
Qwen3.8-max+9.60

Stronger backbones show stronger steering intelligence in the zero-shot setting, with gains of +0.45 (Qwen3-4B), +1.84 (Qwen3-8B), +2.67 (Qwen3.5-9B), and +9.60 (Qwen3.8-max). Due to the high training cost, we are not able to finish a new training run with another backbone in a short time, and we will update the results once we obtain them.


References

[1] Luo et al. DeepCoder: A Fully Open-Source 14B Coder at O3-mini Level. Agentica and Together AI, 2025.

[2] Yang et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS, 2024.

[3] Hanley and McNeil. The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve. Radiology, 1982.

AuthorsOpenReview ↗

Kind Reminder and Additional Results

Dear Reviewer Ms6L,

Thank you again for your detailed review. As the discussion period is closing soon, we would like to kindly check whether our responses have addressed your concerns.

Regarding Q3, we now provide the RL result with a stronger steering backbone. We train Qwen3.5-9B as the steering backbone with the same RL algorithm to further verify the generalizability of our method. This experiment is only partially completed, and we attach the latest intermediate result below:

Steering backboneZero-Shot ΔRL Δ
Qwen3-8B (ours)+1.84+6.66
Qwen3.5-9B+2.67+7.92 (60 steps)

With only 60 training steps, the Qwen3.5-9B steering agent already reaches a gain of +7.92, higher than the Qwen3-8B agent trained for 300 steps (+6.66). This shows that a stronger backbone gives a stronger steering agent in both the zero-shot and RL settings, and further supports that our RL algorithm transfers to other backbones.

We would be glad to clarify any remaining concerns, and would appreciate it if you could consider re-evaluating our work in light of these results.

Sincerely,

The Authors

Thanks for your response. My concerns have been addressed and I have updated my score to 3. Good luck!

We are happy to hear that we've addressed your concerns, and we thank the reviewer again for the feedback!

Reviewer_6CLcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a learning-based steering framework to mitigate goal drift in long-horizon LLM-based single-agent and multi-agent systems. It introduces an external steering agent that supervises a frozen executor during task execution, deciding when to intervene through a lightweight fast path and how to intervene through a reasoning-based slow path. To train the steering agent, the authors develop a reinforcement learning framework that evaluates alternative interventions through executor continuations. The fast path is optimized using a pairwise ranking objective based on the outcome changes caused by interventions, while the slow path is trained using GRPO with an unsteered continuation as an additional reward baseline. Experiments on seven QA benchmarks demonstrate that the RL-trained steering agent improves average F1 in both single-agent and multi-agent settings. Additional experiments investigate out-of-domain generalization, transfer to unseen executors, and controllable intervention frequency.

Critical strengths
  • The paper addresses an important problem in long-horizon agent execution that locally plausible actions can progressively deviate from the overall task objective. The proposed framework is practical as it separates lightweight intervention scheduling from more expensive corrective reasoning, allowing an external supervisor to improve the execution process without updating the executor.
  • The proposed method achieves consistent improvements over the evaluated planning, verification, and untrained steering baselines. The transfer results are also encouraging, with gains on three unseen executor models. These results provide evidence that the learned steering capability is not entirely specific to the training executor.
Critical weaknesses
  • Equation (8) defines steering priority as the mean absolute reward difference between steered and unsteered continuations. In this case, states where interventions consistently degrade performance can receive high priority, encouraging unnecessary or harmful interventions. Similarly, including the unsteered baseline in the GRPO normalization in Equation (10) does not guarantee that positively reinforced interventions actually outperform the no-intervention action. As the experiments primarily measure final task performance rather than directly evaluating goal-drift detection or correction, it remains unclear whether the improvements arise from mitigating goal drift, providing additional task-specific guidance, or other benefits of external supervision.
  • It is unclear whether the gains come from correcting goal drift or from a stronger supervisor injecting task knowledge into a weaker executor. The steering agent (Qwen3-8B) is a later-generation model than every executor it supervises, and Table 3 shows the gain grows as the executor gets weaker (+15.78 for Llama-3.1-8B and +12.01 for Mistral-7B vs. +6.66 for Qwen2.5-7B), which might be attributed to knowledge injection.
Questions for authors
  • Does the gain survive when the steering agent is the same model as the executor, or even a weaker one?
  • The efficiency evaluation largely reports executor turns and intervention counts rather than end-to-end inference cost. Given that the slow path requires approximately 30 seconds per invocation, could the authors also report total inference tokens, wall-clock latency, and overall execution cost, including the steering agent?
  • Could the authors provide additional controlled ablations such as removing the pairwise ranking loss, training only the slow path, or replacing the learned fast path with random or heuristic intervention scheduling under matched intervention budgets to support the proposed design?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6CLc (1/2)

W1: The absolute value in Eq. 8 and the pooled baseline in Eq. 10 may reinforce harmful interventions.

A: We thank the reviewer for pointing out these concerns. We agree that Eq. 8 and Eq. 10 alone do not rule out these cases. However, both are deliberate design choices, which we explain below and validate with an ablation experiment.

  • Eq. 8: By our design, the fast path judges the value of triggering the slow path, following the notion of critical states [1, 2, 3]. Its goal is to find the states where steering has a large impact on the final outcome, while how to steer, i.e., avoiding negative impact and increasing positive impact, is the responsibility of the slow path. In contrast, if we directly encouraged the fast path to select the states where steering currently improves the outcome, the slow path would never get the chance to learn on the states that have a large impact on the outcome but where it has not yet learned to steer well. With this critical-state objective, the steering agent achieves a large gain even at a low steering frequency, where its gain per intervention is the largest (Fig. 4).

  • Eq. 10: For the slow path, we follow GRPO [4] and use a relative learning signal instead of an absolute one, where each steering action is compared with the other outcomes from the same state. Such a group baseline is a standard technique for reducing the variance of policy gradients [5, 6, 7], while the unsteered outcome alone is a noisier reference. Meanwhile, the unsteered baseline is included in the group, so when the steering actions at a state are on average worse than no intervention, the baseline rises and their advantages are lowered accordingly.

In response to the reviewer's concern, we further conduct an ablation with two variants. The Eq. 8 Variant removes the absolute value in Eq. 8 and measures the steering priority by the signed outcome change, i.e., q(s)=1G∑j=1G(Rˉs,j−bs)q(s)=\frac{1}{G}\sum_{j=1}^{G}(\bar R_{s,j}-b_s), so that the states where steering degrades the outcome no longer receive a high priority. The Eq. 10 Variant replaces the relative signal in Eq. 10 with the absolute gain over the unsteered baseline, so that a steering action is positively reinforced only when it outperforms no intervention.

MethodΔ
Ours+6.66
Eq. 8 Variant+5.88
Eq. 10 Variant+5.04

The ablation results validate our design, where both variants fall behind our original objective. We will add this result and the discussion in the revision.


W2 & Q1: Do the gains come from correcting goal drift? Does the gain survive when the steering agent is the same model as the executor, or even a weaker one?

A: We thank the reviewer for pointing out the evaluation gap. To further investigate where the improvement of the steering agent comes from, we randomly sample 100 cases where the steering agent successfully corrects a failure of the executor and ask volunteers to annotate what it corrects in each case. We find that 78 of the 100 cases are caused by correcting goal drift. The remaining cases are those where the steering agent points out outputs in a wrong format and incorrect tool calls. Among the goal-drift corrections, 41 are cases where the executor had retrieved the needed evidence but kept searching, 16 are cases where it pursued an entity or relation the question did not ask about, 6 are cases where it kept repeating the same query, and the rest belong to other types. This result shows that the steering agent behaves as expected and improves the performance mainly by correcting goal drift.

In response to the reviewer's question, we further replace the Qwen2.5-7B executor with Qwen3-8B, Qwen3-14B, and Qwen3.5-4B and apply the same steering agent (Qwen3-8B as backbone trained with Qwen2.5-7B as the executor) without retraining:

MethodQwen2.5-7B Avg.ΔQwen3-8B Avg.ΔQwen3-14B Avg.ΔQwen3.5-4B Avg.Δ
No Intervention42.70–51.60–58.40–58.47–
Zero-Shot Steering44.54+1.8452.41+0.8157.87−0.5359.05+0.58
RL Steering49.36+6.6655.54+3.9460.09+1.6960.36+1.89

When steering the same model, the RL-trained steering agent still improves it by +3.94. When steering stronger models, the gain is smaller but still positive (+1.69 on Qwen3-14B and +1.89 on Qwen3.5-4B).

The gain on stronger executors is smaller than on Qwen2.5-7B, since stronger executors are less prone to goal drift on the search tasks we tested. However, we expect the steering paradigm to bring more benefit to strong executors when tasks get harder, such as GAIA. To verify this, we evaluate the Qwen3.5-4B executor on GAIA, where the steering agent improves the performance from 27.18 to 32.52, an improvement of around 20%. Due to the limited search API budget, we leave the exploration of more complex tasks to future work.

AuthorsOpenReview ↗

Response to Reviewer 6CLc (2/2)

Q2: End-to-end inference cost, including the steering agent.

A: We compare the execution agent alone, the steering agent solving the task alone, and our system:

MethodAvg.Input tokensOutput tokensLatency
Execution agent only42.7017.8k0.1k3.6 s
Steering agent only49.9711.5k5.1k85 s
Execution agent + steering agent (ours)49.3638.3k1.8k34 s

The execution agent alone is efficient, taking 3.6 s per task, but it only reaches 42.70. The steering agent alone improves the performance to 49.97, while its latency rises to 85 s. Our system keeps the performance at a comparable level (49.36) and reduces the latency to 34 s, 60% lower than the steering agent alone. Note that the additional input tokens of our system come from the steering agent re-reading the trajectory at each judgment, and input tokens are processed in a single prefill pass at low cost (about 0.2 s per call, Appendix B.2). The latency is dominated by output tokens.


Q3: Additional controlled ablations on the fast path.

A: We thank the reviewer for suggesting this ablation. We supplement the ablations below under matched intervention budgets. For heuristic scheduling, we adopt entropy, a commonly used indicator that reflects the uncertainty of the model at the current step:

VariantAvg.Δ
Random scheduling46.75+4.05
Entropy-based scheduling48.62+5.92
Fast-slow path Steering (ours)49.36+6.66

The results show the effectiveness of the learned fast path, which outperforms both random scheduling and entropy-based scheduling (+6.66 vs. +4.05 and +5.92). We will add this ablation in the revision.

Due to the high cost of training, we are not able to finish the training without the ranking loss in a short time. Nonetheless, we believe that the results above already demonstrate the effectiveness of the fast path trained with the ranking loss.


References

[1] Shen et al. CARL: Focusing Agentic Reinforcement Learning on Critical Actions. arXiv:2512.04949, 2025.

[2] Huang et al. Establishing Appropriate Trust via Critical States. IROS, 2018.

[3] Amir and Amir. HIGHLIGHTS: Summarizing Agent Behavior to People. AAMAS, 2018.

[4] Shao et al. DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300, 2024.

[5] Williams. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Machine Learning, 1992.

[6] Kool et al. Buy 4 REINFORCE Samples, Get a Baseline for Free! ICLR Workshop on Deep RL Meets Structured Prediction, 2019.

[7] Ahmadian et al. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. arXiv:2402.14740, 2024.

AuthorsOpenReview ↗

Kind Reminder and Additional Results

Dear Reviewer 6CLc,

Thank you again for your detailed review. As the discussion period is closing soon, we would like to kindly check whether our responses have addressed your concerns.

Regarding Q3, we have now completed the ablation study without the ranking loss. This variant removes the pairwise ranking loss and trains only the slow path:

VariantAvg.Δ
w/o ranking-loss training48.14+5.44
Fast-slow path Steering (ours)49.36+6.66

The results show that removing the ranking loss and training only the slow path leaves the model unable to judge which states have a large impact on the outcome. As a result, the gain brought by steering drops by 1.22 points (+5.44 vs. +6.66).

We would be glad to clarify any remaining concerns, and would appreciate it if you could consider re-evaluating our work in light of these results.

Sincerely,

The Authors

Dear Reviewer 6CLc,

We understand this is a very busy period, and we apologize for following up again. As the discussion period closes in less than one day, we would be grateful for your feedback on whether our responses have addressed your concerns. We would be glad to clarify any remaining questions before the discussion closes.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.