- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how coding agents can better follow instructions and constraints from multiple sources during task execution. It introduces Applicable Constraint Retrieval (ACR), which learns to select relevant constraints using the interaction history and the proposed action. ACR is integrated into Applicable Constraint Monitoring (ACONM), which checks actions before execution and provides feedback for revision when violations are detected. Experiments show improved coverage of applicable constraints and better constraint compliance across three models on OctoBench. An additional evaluation on SWE-Gate provides preliminary evidence of transfer without retraining the retriever.
Critical strengths
- The paper addresses a practical problem with a reasonable solution. Coding agents can overlook or violate constraints while carrying out tasks. The proposed approach retrieves relevant constraints using the interaction history and proposed action, checks them before execution, and provides feedback for revision. This design directly addresses reliability requirements in practical coding tasks.
- The evaluation provides substantial support for both retrieval quality and the complete system. With the same inputs and retrieval budget, ACR improves applicable-constraint coverage from 56.90% to 66.96% and maintains an advantage across different retrieval cutoffs. The bootstrap analysis in the appendix further supports this improvement, while human annotation provides evidence for the quality of the supervision labels. The complete system improves constraint compliance across three models on OctoBench and shows preliminary transfer results based on executable tests on SWE-Gate.
Critical weaknesses
- The specific contribution of learned retrieval to downstream performance remains to be clarified. The offline experiments provide convincing evidence that training improves constraint retrieval, whereas the online experiments evaluate the complete system, including checking, feedback, and revision. A comparison between ACR and the pretrained retriever within an otherwise identical monitoring pipeline would help establish how much the retrieval improvement translates into better final constraint compliance.
Questions for authors
- How do final constraint compliance and task success change when ACR is replaced with the pretrained retriever while keeping the monitoring and revision procedures fixed?
- Relative to Vanilla, what are the average additional model calls, token usage, and end-to-end runtime?
- How would rerunning ACR after each revision, rather than reusing the original retrieval results, affect constraint compliance and computational cost?