acceptodds
Under review as a conference paper at ICLR 2027

Learning to Select Constraints for Coding Agents

Abstract

Coding agents must follow written constraints on how they implement, test, and report their work. Different constraints apply at different steps, so a monitor must decide what to check before each action. We use OctoBench, a benchmark that provides coding tasks, their written instructions, and a checklist of requirements for each task. The task instructions provide the text to retrieve, and the checklists specify the requirements used to evaluate constraint following. We propose Applicable Constraint Retrieval (ACR), which learns to select instructions according to the proposed action and earlier interactions. We further design Applicable Constraint Monitoring (AConM), which uses the retrieved instructions to check proposed actions and request revisions when needed. On OctoBench, ACR retrieves instructions that fully express 66.96% of applicable constraints on average across tasks, compared with 56.90% for a pretrained retriever given the same inputs and budgets. On OctoBench, 85.34% of requirements are marked as met with AConM, compared with 79.72% without monitoring, averaged equally across three coding models. In an exploratory evaluation on 50 SWE-Gate tasks, we reuse the same retriever without further training. With DeepSeek-V4-Flash, 38% of tasks pass both functional and constraint tests with monitoring, compared with 32% without it. The source code is available at https://anonymous.4open.science/r/aconm-3219.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, multi-turn constraint following, coding agents, turn-level constraint relevance, constraint selection, runtime monitoring, automated repair

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gsojOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how coding agents can better follow instructions and constraints from multiple sources during task execution. It introduces Applicable Constraint Retrieval (ACR), which learns to select relevant constraints using the interaction history and the proposed action. ACR is integrated into Applicable Constraint Monitoring (ACONM), which checks actions before execution and provides feedback for revision when violations are detected. Experiments show improved coverage of applicable constraints and better constraint compliance across three models on OctoBench. An additional evaluation on SWE-Gate provides preliminary evidence of transfer without retraining the retriever.

Critical strengths
  1. The paper addresses a practical problem with a reasonable solution. Coding agents can overlook or violate constraints while carrying out tasks. The proposed approach retrieves relevant constraints using the interaction history and proposed action, checks them before execution, and provides feedback for revision. This design directly addresses reliability requirements in practical coding tasks.
  2. The evaluation provides substantial support for both retrieval quality and the complete system. With the same inputs and retrieval budget, ACR improves applicable-constraint coverage from 56.90% to 66.96% and maintains an advantage across different retrieval cutoffs. The bootstrap analysis in the appendix further supports this improvement, while human annotation provides evidence for the quality of the supervision labels. The complete system improves constraint compliance across three models on OctoBench and shows preliminary transfer results based on executable tests on SWE-Gate.
Critical weaknesses
  1. The specific contribution of learned retrieval to downstream performance remains to be clarified. The offline experiments provide convincing evidence that training improves constraint retrieval, whereas the online experiments evaluate the complete system, including checking, feedback, and revision. A comparison between ACR and the pretrained retriever within an otherwise identical monitoring pipeline would help establish how much the retrieval improvement translates into better final constraint compliance.
Questions for authors
  1. How do final constraint compliance and task success change when ACR is replaced with the pretrained retriever while keeping the monitoring and revision procedures fixed?
  2. Relative to Vanilla, what are the average additional model calls, token usage, and end-to-end runtime?
  3. How would rerunning ACR after each revision, rather than reusing the original retrieval results, affect constraint compliance and computational cost?
Reviewer_m87uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies which written requirements should be selected for checking before a coding agent executes an action. The authors propose ACR, a constraint retrieval method that uses the interaction history and proposed action to retrieve relevant instructions. The retriever is trained on LLM-generated applicability labels, including requirements that apply to an action but are already satisfied. Building on ACR, the AConM monitoring system checks actions for violations and requests revisions when needed. The experiments show that ACR improves constraint retrieval coverage on OctoBench, while the complete system improves constraint following across three coding models. An exploratory evaluation on SWE-Gate also shows an improvement in the joint pass rate for functional and constraint tests.

Critical strengths
  1. The paper addresses a practical problem. Different requirements need to be checked when a coding agent reads files, modifies code, or sends its final response. Using this action-dependent applicability as the retrieval objective, including requirements that are already satisfied, offers a useful perspective on instruction following.

  2. The retriever's improvement is supported by a direct comparison. With the same inputs and number of retrieved candidates, ACR improves constraint coverage from 56.90% to 66.96%. The complete system improves constraint following across three acting models, and the executable tests on SWE-Gate provide evidence beyond LLM-based scoring.

Critical weaknesses
  1. I would like to better understand the trained retriever's contribution to final performance. The paper already demonstrates improved retrieval coverage, but the complete system also includes applicability checks, violation checks, and multiple revision attempts. Comparing the pretrained and trained retrievers while keeping these stages unchanged would help clarify how much retriever training improves final task performance. Since learning to select constraints is the paper's main contribution, this comparison is important for assessing the method's value. (Sections 4 and 5)

  2. I would appreciate further explanation of why some comparison methods perform substantially worse after monitoring is added. For example, on SWE-Gate, 37 tasks pass the functional tests without monitoring, compared with 26, 28, and 24 for Interwhen, AgentSpec, and ShieldAgent, respectively. It is not yet clear whether these drops mainly come from incorrect checks, errors introduced during revision, execution-budget limits, or differences introduced when adapting the methods. Some failure cases and further details about the adaptations would help readers interpret these results. (Tables 1 and 2)

Questions for authors
  1. Could the authors compare the pretrained and trained retrievers while keeping the checking model, prompts, number of retrieved candidates, and revision rules the same, and report the differences in final constraint satisfaction and task success rates?
  2. What are the main reasons that several monitoring baselines perform worse than the unmonitored agent? Could the authors provide representative failure cases and explain the adjustments made to apply these methods to the tasks in this paper?
Reviewer_LH3yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Applicable Constraint Retrieval (ACR) and Applicable Constraint Monitoring (AConM) to improve constraint adherence in coding tasks. ACR informs AConM by retrieving a set of candidate constraints for each action, which are then checked for applicability and violation. AConM outperforms other baselines in constraint adherence for coding. Additionally, ACR can improve constraint monitoring on other benchmarks without retraining.

Critical strengths
  • Outperforms prior methods for coding agent constraint adherence

  • Evaluates transferability to other tasks without retraining, strengthening results

Critical weaknesses
  • The paper is missing an important baseline for establishing the effectiveness of their method. Namely, if all the constraints were returned from the retrieval step, how would AConM perform? Is ACR a necessary component?

  • Separating retrieval (ACR) and applicability (check in AConM) undermines the motivation of Applicable Constraint Retrieval. While an additional step of applicability verification may be necessary, ACR should instead be presented as candidate constraint retrieval throughout the paper.

Questions for authors
  • How would a baseline implementation where all constraints are retrieved perform?

  • “The learned retrieval result Rt is reused across these attempts, so a revised action may require constraints absent from that set.” - is this the cause of most of the failure cases or are failures for other reasons?

  • The results with the alternative baselines all underperform vanilla. Is that consistent with prior findings? Or an artifact of your problem setup?

  • What is the size of the constraint pool? Is K=50 a significant portion of this pool?

  • “In the SWE-Gate deployment, additional retrieval based on the proposed action can add candidates before applicability and violation checking (Section 4.3). These candidates are checked alongside Rt; Rt denotes only the learned retriever’s output.” - what is this “additional retrieval”? How dependent are results on the additional retrieval compared to Rt alone?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.