acceptodds
Withdrawn from ICLR 2027

When Agents Refuse to Ask for Help: Privileged Self-Distillation for Help-Seeking Software Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language-model agents increasingly execute long-horizon software-engineering tasks, yet some tasks require information present neither in the issue text nor anywhere in the repository. Exposing an ask_human tool does not by itself induce reliable help-seeking. Using HiL-Bench, we first localize the failure by inserting two diagnostic setups between \askhuman and \fullinfo that withhold resolutions and supply only a cue about what is missing. \askdescription recovers and of the \askhuman to \fullinfo Pass@3 gap for GLM-5.2 and Qwen3.8-27B, while \askhint recovers and . Identifying which information is missing, rather than acting on it once supplied, is therefore the dominant measured bottleneck. We then ask whether cue-elicited behaviour transfers to a model that receives no cue at inference time. We construct \dataset, a corpus of blocker-augmented tasks built from SWE-ReBench V2 by rewriting only the observable task specification while leaving the repository, revision, test suite and target patch unchanged. We introduce Privileged Self-Distillation (\psd), in which a teacher rolls out with privileged blocker cues, the cues are removed, and an unprivileged student initialized from the same weights is trained on the resulting trajectories. We study a matrix of privileged cue source (\askhint versus \askdescription) and supervision policy (\askall versus \askclean, which masks the training loss on judge-rejected asking turns while leaving those turns in context). All four recipes fine-tune Qwen3.8-27B with reasoning enabled. \descclean attains the highest task success (Pass@1 , Pass@3 ) and the highest blocker recall (), whereas \descall attains the highest question precision () and Ask-F1 (). Cleaning raises task success and recall under both cue sources, and lowers precision and Ask-F1 under both.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Human-in-the-Loop, Software Agents, Help-Seeking, Privileged Distillation, Self-Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NjAnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates why software agents fail to seek help when task-critical information is unavailable. Diagnostic experiments on HiL-Bench show that providing hints or descriptions of missing information, without revealing the answers, substantially narrows the performance gap between agents with access to a help tool and fully informed agents. The paper also introduces SWE-ReBench-V2-Blocked and proposes privileged self-distillation (PSD), training a student on trajectories generated by the same initial model with privileged cues removed from the student’s input. Four training recipes vary the cue source and whether judge-rejected asking targets are excluded from the loss. Description-Clean achieves the highest mean task success, whereas Description-All achieves the highest question precision and Ask-F1. The authors do not establish protocol equivalence between the diagnostic experiments and the PSD evaluation, precluding direct comparisons between their results.

Critical strengths
  1. The Hint and Description interventions provide informative intermediate conditions between receiving no cue and receiving the answers. Evaluations with two coding models and executable tests provide useful evidence for diagnosing help-seeking failures within the HiL-Bench setting.

  2. The paper reports all four training recipes, making the trade-off between task success and question quality visible. It also explicitly acknowledges limitations in protocol comparability, supervision dose, judge reliability, and task construction.

Critical weaknesses
  1. Missing baseline and attribution. The paper lacks a protocol-matched untrained baseline and an ordinary SFT comparison, leaving the net benefit of PSD and the contribution of privileged cues unclear.

  2. Insufficient corpus validation. The constructed tasks lack corpus-wide solvability checks and human audits of blocker recoverability, task naturalness, and information leakage, raising concerns about supervision quality.

  3. Limited evidence for broader generalization. PSD is trained on synthetically blocked SWE-ReBench V2 tasks and evaluated on 100 public HiL-Bench tasks. While this tests transfer to HiL-Bench, robustness across other task constructions remains unestablished.

Questions for authors
  1. Can the authors evaluate the untrained agent and an ordinary SFT baseline under the same protocol as PSD to isolate its contribution and quantify the resulting trade-offs?

  2. What fraction of a representative sample becomes solvable when all registered answers are supplied, and can an audit verify the validity, recoverability, and naturalness of the constructed tasks?

Reviewer_6h4qOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies coding agents faced with requirements that omit information unavailable in the repository. It uses HiL-Bench to compare agents, given either only asking tool, or asking tool and hints about the missing information, or the tool with descriptions of the missing information. They used Llama-3.3-70B-Instruct-AWQ as the judge of the questions and tested it on two LLM models, Qwen3.8-27B and GLM-5.2. The description does not provide any answers, but recovers most of the gap that was present compared to the model with only the tool call provided. The authors interpret this as evidence that identifying the information gap is a major obstacle in completing a task. The authors then construct SWE-ReBench-V2-Blocked dataset, which contains 5600 tasks and 19164 blockers, which are essentially missing information. They then introduce Privileged Self-Distillation (PSD). Here, they collect trajectories given clues about the blocker, then remove the hint and fine-tune a student initialized from the same model with the questions and the corresponding answers. They do experiments on 4 different combinations, by combining hint or description-based clues with either trajectory, with all questions (both accepted and rejected by the judge model) called All, or only accepted questions called Clean. Description-clean has the highest task success, while description-all has the highest question precision and ask-f1.

Critical strengths
  • The experiments done to diagnose the problems are strong. They were able to clearly show that blocker descriptions can increase the success rate at Pass@3 from 26.4% to 76.0% for GLM-5.2 and from 23.0% to 81.2% for Qwen3.8-27B. Knowing what information is missing helps significantly to nudge the model to ask questions about missing information, compared to just providing a tool that permits asking questions. While this is good, they did not isolate the reason, could it have been a result of prompts that were not optimized etc.
  • Reporting interaction quality instead of just task success. Task success determines if the agent solves the task, while the LLM judge checks whether its questions target the specific blockers. Precision penalizes irrelevant or repeated questions, and recall measures the blocker coverage. These show an interesting outcome. Description-Clean has a higher pass@1 than Description-All but lower precision. Reporting both shows that more tasks are done necessarily means asking better questions.
Critical weaknesses
  • The first major weakness is a lack of control/untrained baseline. They compare 4 trained models but don’t really show whether training made the original model better. They do a separate comparison before they start the training, but then mention that this is not really a fair comparison, which is correct. But that begs the question, did PSD help? Even if PSD helps, one other thing that could have been done is ordinary fine-tuning without hints. That could show whether the benefit comes specifically from their method or just because of fine-tuning.
  • The other major issue is that the prior work section misses several relevant papers (Snell et al., Zelikman et al.). They both make the model generate useful behavior, then train it to produce behavior without original help. I would consider this a major gap in positioning and related work. In this context, the only novelty is where this technique is applied.
Questions for authors
  • Can you provide untrained Qwen3.8-27B results under the exact Table 4 evaluation settings, and clarify which settings differ from Section 3? Do you have a comparison with fine-tuning without the clues?
  • How does PSD differ from context distillation and STaR, outside the difference in the information supplied and the application setting? Which experiments show why this difference matters?
Reviewer_cRLnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper discusses how an agent should ask clarification questions in software agent settings. Beyond that, I was not able to get better understanding on what exactly the problem is and how it is solving it.

Critical strengths

None. I cannot understand the majority of the paper.

Critical weaknesses

Paper is too poorly written for a human to review. See below for some of these cases. As such, I do not understand the majority of what the paper is discussing.

Questions for authors

Intro is so poorly written (I'd suspect by AI). For instance, impossible to understand as part of intro:

We separate blocker identification from post-identification recovery, the work of formulating a question and using the answer. Two inserted setups withhold resolutions and supply only a cue, AskHuman-Hint a short abstract one and AskHuman-Description the full blocker description

What are these terms? What is "inserted setup"?

In 2.1, "removes task-critical information from s tilta". Do you mean s here? What is a blocker? These terms are just not defined before they are introduced.

Things are just not defined... What;s the difference between AskHuman-Hint/AskHuman-Description ?

I cannot understand paragraphs like this, but I'd really be curious if you can help me unpack what this paragraph means.

A salient blocker lets the same base model ask a relevant question and solve the task, so the deficit depends on the agent’s information state. The cues alter context, so we do not establish that the unprivileged policy is already latent. The trained arms are not run under this protocol, so we draw no comparison with the numbers here.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.