- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates why software agents fail to seek help when task-critical information is unavailable. Diagnostic experiments on HiL-Bench show that providing hints or descriptions of missing information, without revealing the answers, substantially narrows the performance gap between agents with access to a help tool and fully informed agents. The paper also introduces SWE-ReBench-V2-Blocked and proposes privileged self-distillation (PSD), training a student on trajectories generated by the same initial model with privileged cues removed from the student’s input. Four training recipes vary the cue source and whether judge-rejected asking targets are excluded from the loss. Description-Clean achieves the highest mean task success, whereas Description-All achieves the highest question precision and Ask-F1. The authors do not establish protocol equivalence between the diagnostic experiments and the PSD evaluation, precluding direct comparisons between their results.
Critical strengths
-
The Hint and Description interventions provide informative intermediate conditions between receiving no cue and receiving the answers. Evaluations with two coding models and executable tests provide useful evidence for diagnosing help-seeking failures within the HiL-Bench setting.
-
The paper reports all four training recipes, making the trade-off between task success and question quality visible. It also explicitly acknowledges limitations in protocol comparability, supervision dose, judge reliability, and task construction.
Critical weaknesses
-
Missing baseline and attribution. The paper lacks a protocol-matched untrained baseline and an ordinary SFT comparison, leaving the net benefit of PSD and the contribution of privileged cues unclear.
-
Insufficient corpus validation. The constructed tasks lack corpus-wide solvability checks and human audits of blocker recoverability, task naturalness, and information leakage, raising concerns about supervision quality.
-
Limited evidence for broader generalization. PSD is trained on synthetically blocked SWE-ReBench V2 tasks and evaluated on 100 public HiL-Bench tasks. While this tests transfer to HiL-Bench, robustness across other task constructions remains unestablished.
Questions for authors
-
Can the authors evaluate the untrained agent and an ordinary SFT baseline under the same protocol as PSD to isolate its contribution and quantify the resulting trade-offs?
-
What fraction of a representative sample becomes solvable when all registered answers are supplied, and can an audit verify the validity, recoverability, and naturalness of the constructed tasks?