- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies when to intervene before a coding agent acts. It proposes HiSentinel, a small model that allows an action, redirects the agent, or asks for human input. During training, a teacher uses recorded future outcomes to provide selected training targets. At deployment, HiSentinel sees only the current context and proposed action. The paper introduces SWE-INTERVENE and tests intervention detection and task success across three coding agents.
Critical strengths
- The method improves code repair and clarification results with Qwen3-Coder, Devstral, and Sonnet-4.6. This suggests that its benefits extend across different coding agents.
- Table 4(c) directly tests the value of hindsight supervision. At similar token costs, the full method reaches 44% task success, versus 36% with a teacher that cannot see future outcomes and 34% without distillation. This supports the main training idea.
Critical weaknesses
- The online evaluation covers only 50 tasks on SWE-bench Verified Mini and 100 on Ask or Assume. In Table 4(c), the gain over the teacher without future information is four successful tasks. Without repeated runs or uncertainty estimates, it is hard to judge how stable this gain is.
- The value of the three-way intervention design is less clear. Prior work already trains small critics with SFT and DPO [1] and uses privileged feedback for learning [2]. Table 4 supports distillation, but does not separately test HARD-PAUSE or feedback DPO online. Their benefit over a simpler corrective critic remains unclear, especially on Ask or Assume.
References: [1] Shubham Gandhi et al. Steer, Don’t Solve: Training Small Critic Models for Large Coding Agents. 2026. arXiv:2606.21811. [2] Sanjiban Choudhury and Paloma Sodhi. Better than Your Teacher: LLM Agents that Learn from Privileged AI Feedback. ICLR 2025. arXiv:2410.05434.
Questions for authors
- Can you report paired confidence intervals or repeated runs for Table 4(c), especially for the full method versus the teacher without future information?
4 comments
Dear Reviewer,
Thank you for taking the time to review our paper. Your feedback has helped us identify where the paper could be clearer. As the discussion period draws to a close, we would appreciate it if you could read our rebuttal and let us know whether it addresses your concerns. We would be happy to clarify any remaining points.
Best regards, The Authors
Dear Reviewer,
Thank you for taking the time to review our paper and share your feedback.
If our rebuttal has addressed your concerns, we would be very grateful if you would consider raising your rating. Your support would mean a great deal to us. If you have any further questions, we would be happy to answer them.
Thank you again for your time and consideration.
Best regards,
The Authors
We Would Be Grateful for Your Feedback on Our Rebuttal
Dear Reviewer drDU,
Thank you for taking the time to review our paper and share your feedback. After seeing that your review was updated on November 17 at 17:33, we wanted to follow up on our rebuttal. We would be very grateful if you could let us know whether the additional experiments and explanations address the concerns you raised.
If you feel that they do, we sincerely hope you would consider raising your rating. Your support for this work would mean a great deal to us. If you have any remaining questions, we would be very happy to answer them.
Thank you again for your time and consideration.
Best regards,
The Authors
We sincerely appreciate your thoughtful review and your recognition of HiSentinel’s gains across coding agents and the value of hindsight supervision.
Evaluation scale and stability. Unfortunately, our compute budget did not allow us to run all 500 tasks for every experimental setting. We therefore used the previously published Verified Mini subset and chose greedy decoding (
do_sample=False, with temperature set to 0) to minimize randomness in the original evaluation. We understand your concern about the stability of the results and used our available compute for the following two experiments.First, we evaluated our main comparison on all 500 tasks in SWE-bench Verified: Qwen3-Coder-30B-A3B with and without the 1.7B HiSentinel, with all other evaluation settings unchanged.
Although the percentage-point gain is smaller than on Verified Mini, HiSentinel still achieves a substantial improvement in resolution rate on the full 500-task benchmark.
Second, in response to your request, we repeated the ablation study on SWE-bench Verified Mini. We ran each setting three times with sampling enabled (
do_sample=True,temperature=0.2), using seeds 42, 43, and 44, and report the average across the three runs.Although these experiments still use the Mini subset, the repeated runs provide further evidence that the gains from our method are stable.
Three-way intervention design. At each proposed action, we first decide whether to intervene. If intervention is needed, we then decide whether the coding agent can address the problem on its own or needs human assistance. These two decisions give rise to the three routes defined in Table 1: ALLOW when no intervention is needed, REDIRECT when the agent can correct the issue autonomously, and HARD-PAUSE when human assistance is necessary.
We believe these routes have clear boundaries and reflect how coding agents are used in practice. Together, they cover the main situations an agent encounters at this decision point. We therefore view HARD-PAUSE as part of a three-way design that is essential to our methodology, alongside ALLOW and REDIRECT, rather than as an add-on or a standalone component. For this reason, we do not consider it appropriate to remove HARD-PAUSE for a separate test.
Relation to prior work. We agree that SFT followed by DPO is an established training recipe and that the value of privileged information has been demonstrated in prior work. In our view, the distinction lies in how these techniques are used and what they are used to learn. In HiSentinel, we use SFT and DPO to train feedback generation. For this purpose, we follow a standard, widely used training recipe without proposing any changes to SFT or DPO themselves. We see our main contribution as formulating the intervention decision at the point between action proposal and execution, and distilling a privileged teacher’s intervention judgments into a lightweight sentinel. We believe these contributions, taken together, are both substantial and novel.
We appreciate the comparison with Steer, Don’t Solve [1], which we include as SDS in Table 2 as a baseline. Our best HiSentinel (1.7B) outperforms SDS’s best model (4B) [1] by 9.5 pp on average.
As we understand it, LEAP [2] puts privileged information directly into the prompt used by the teacher to generate feedback. The two approaches also differ in both what they train and how they train it: LEAP fine-tunes the acting agent on the teacher’s corrections, whereas we distill the teacher’s hindsight-informed judgments into a separate sentinel that makes pre-execution intervention decisions. Beyond this specific comparison, we discuss the broader use of privileged information in the second paragraph of Related Work. We believe that privileged information is widely used, and that the novelty of our approach lies in how we use it. In our view, this is also what most clearly distinguishes our approach from prior work. We appreciate you bringing this paper to our attention and will cite it in that paragraph of Related Work.
Thank you again for your time and feedback. If the additional experiments and explanations address your concerns, we would be grateful if you would consider raising your rating. We welcome any further questions and will do our best to answer them.