acceptodds
Withdrawn from ICLR 2027

Beyond Instruction Following: Learning Grounded Skill-Following with Skill Contracts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Instruction following typically enforces discrete, response-level requirements, whereas an expert-authored skill prescribes procedural requirements spanning multiple phases and environment interactions. Given such a skill, we aim to train the executor to execute all required phases successfully instead of focusing solely on the final answer. We therefore introduce Grounded Skill-Following, which requires an agent to execute a fixed, expert-authored skill across its required phases by grounding decisions in environment observations. To achieve verifiable procedural execution, we formulate each skill as a skill contract combining visible skill instructions with an explicit contract runtime. The contract runtime specifies the required phases, admissible actions, permitted transitions, and accepted termination. This contractual structure naturally provides a dense, verifiable training signal throughout execution. We leverage this by introducing Verified Progress Credit, which assigns rewards upon the initial completion of contract milestones and aggregates them into the trajectory return to guide policy optimization. During rollout, the contract runtime continuously tracks state transitions to provide Contract-State Feedback, which indicates whether the latest action is accepted and guides the agent toward valid next actions. To measure procedural compliance, we introduce the Protocol Completion Rate (PCR)—defined as reaching accepted termination through all required phases—and decouple it from the final Task Outcome. Jointly trained with our framework, Qwen3.5-4B achieves Protocol Completion Rates of 99.27% on Math and 99.96% on Search, while slightly outperforming original baselines in Task Outcome (82.95% and 46.61%, respectively). Controlled studies examine how skill instructions, training signals, and contract-state feedback affect both metrics, while withholding interventions evaluate behavioral dependence on observation content.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, LLM Agents, Tool Use, Instruction Following, Process Reward, Reward Shaping

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_WeUWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines grounded skill-following: an agent has to run a fixed, expert-written skill through all of its required phases. Each skill becomes a "skill contract", meaning visible instructions plus a hand-written state machine. The state machine checks every action, sends next-step hints, and adds first-time milestone rewards to the task reward during GRPO training. On Math and Search with five Qwen executors, trained policies reach 94–99.99% Protocol Completion Rate. Task Outcome is roughly on par with matched ReTool/Search-R1 recipes. The paper also runs credit/feedback ablations and inference-time interventions on the skill text, observations and feedback.

Critical strengths
  1. Compliance and correctness both come from one deterministic verifier and are always reported separately. The main text is honest about results that go against the method: no consistent gain over Recipe, ρ=1.0 beating ρ=0.9 on outcome, and heavy reliance on feedback. The method itself is simple, so these findings are what make the paper useful.
  2. The Recipe baselines look fair. They share data, retriever and evaluator with the method. Their Search numbers line up with the published Search-R1 GRPO results (about 32.7 vs 31.2 average EM on Qwen2.5-3B, 37.1 vs 35.0 on 7B). That makes "on par with Recipe" believable across five executors.
Critical weaknesses
  1. The near perfect PCR is largely built into the contract runtime. Inadmissible actions are blocked, and every step comes with Contract-State Feedback that names an admissible next action. When feedback is removed at inference, Search PCR falls sharply, as Panel B of Table 2 shows. No baseline gets a compliance score, since PCR for Recipe is marked n/a in Table 1. The paper's motivation is that outcome-driven RL methods such as Search-R1 [1] or ReTool [2] do not follow the procedure. This is never checked with the same contract verifier, not even on the phases that both setups share.
  2. The abstract overclaims. It states that jointly trained Qwen3.5-4B slightly outperforms the original baselines in Task Outcome, but on Math the Recipe baseline scores higher on that same model. Across Table 1, Math results are tied with or worse than Recipe overall. The clear gains appear only on Search with the Qwen3.5 models, and all results come from single training runs.
  3. Both skills share the same phase set, as Table 3 shows. The checks only look at form, such as a minimum length for a substantive think step or non-empty stdout from the interpreter. This is close to checking the action tag format of the Search-R1 [1] and ReTool [2] loops. Two skills of this kind say little about expert-authored skills in general.
  4. The contract runtime is basically a manually written reward machine [3]. The paper argues that its skill contract differs from reward machines because reward machines require rigid state definitions, and this contrast does not hold. Verified Progress Credit is also close to the format and step-wise rewards used in ToolRL [4] and StepSearch [5]. SOPBench [6] already separates compliance with standard operating procedures from outcome by using rule-based verifiers. None of ToolRL, StepSearch, or SOPBench is cited.

[1] Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. [2] ReTool: Reinforcement Learning for Strategic Tool Use in LLMs. [3] Using Reward Machines for High-Level Task Specification and Decomposition in Reinforcement Learning. [4] ToolRL: Reward is All Tool Learning Needs. [5] StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization. [6] SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints.

Questions for authors
  1. In the Prompt rows, Qwen2.5-3B-Instruct gets 75.07% Search PCR but only 0.19% Task Outcome. That is far below its own no-retrieval Direct score of 11.37%. Qwen3.5-4B also falls from 55.44 to 0.83 on Math. Is this an extraction or runtime issue? If untrained models basically cannot use the contract, that changes how I read the expert-authored skill framing.
  2. With a recovery coefficient of 1.00, Math RF-PCR is 0.00 while PCR is 96.23, so every completed trajectory had at least one rejected action. What are those rejected actions? Since rejections cost nothing under this setting, is the policy testing the runtime with rejected actions on purpose?
  3. Was Credit-only evaluated with feedback off? If so, it reaches 98.69 Search PCR without feedback, while Feedback + Credit drops to 43.05 once feedback is removed. What is the case for keeping feedback in the main method?
  4. In Table 9, on Search, swapping in the Math skill instructions costs only 0.15 points, while an independently written paraphrase costs 3.40. How does this fit the claim that execution depends on the visible skill instructions?
  5. Under the same runtime, does the task reward plus a single binary bonus for reaching DONE train as well as your method? The base model almost never reaches DONE, so Outcome-only staying at zero is expected. It is unclear what Verified Progress Credit adds beyond any reward shaping at all.
  6. In Table 2, the Math Task Outcome for a recovery coefficient of 1.00 is 64.61, which is identical to the single-task Math result in Panel B. The Search PCR for a recovery coefficient of 0.90 is 98.90, which is identical to the single-task Search PCR. These are different policies. Can you confirm these are not copy errors?
Reviewer_61dwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to train a policy (executor) to execute different phases in a skill. For this, they create a "skill-contract," which is essentially skill instructions + contract runtime, which includes required phases, actions the policy can take, permitted transitions, and termination requirements. This framework gives denser rewards, and the authors claim to solve the issue of sparse rewards by using "verified progress credit," which gives reward upon the completion of contract milestones. The framework also includes a state feedback mechanism that guides the agent towards the next valid action. For evaluation, 5 different Qwen models are tested on 2 domains, math and search. The authors claim that outcome correctness alone is insufficient to measure procedural compliance, so they decouple compliance from accuracy. They use a new metric called PCR - how often the contract enters the DONE state with the help of feedback and also introduce RF-PCR in addition to PCR and task accuracy. The skill contracts are hand-created, one per domain, and the policy is trained via GRPO against ReTool, Search-R1, prompt, and direct baselines. The paper also includes some ablation/analysis. However, I have several concerns about the technical validity of the framework and the experimental setup.

Critical strengths
  1. The skill-contract pipeline is well engineered. The different phases, milestones, transitions, feedback, etc during runtime, as well as the handling of the rho^h milestone discount, are intuitive.
  2. The motivation for skill-following as a source of dense reward addresses a real problem of GRPO-based methods.
  3. Good ablation/analysis (RQ2, RQ3) -- answers a few questions a reader might encounter while reading the formulation.
Critical weaknesses
  1. The paper is about procedural skill compliance, but the paper never argues why this is worth achieving. I did not find any motivation or argument for why skill following matters more than task completion. How useful is skill following if the agent fails to solve the task?
  2. My biggest concern is the evaluation setup. The same runtime is used during both training and eval. It grades compliance with the same spec the policy used during RL. A fairer eval comparison between the baseline and the proposed method would be to evaluate the trained policy without the skill runtime contract by just giving the skill in the prompt, as the runtime during eval might guide the agent towards the valid next step. As seen in Table 2, B: feedback removal. For search, the drop is almost 17% in task outcome and 55% on PCR. This shows the feedback was doing the heavy lifting.
  3. It would be more interesting and relevant to see feedback-off as the default evaluation setup and not just an intervention experiment. The policy was trained with feedback enabled, so the feedback-off results show the dependence on the guidance rather than a learned behavior or compliance.
  4. Does the recipe implementation also use only an outcome reward? If so, Table 1 shows that the proposed method does not consistently beat the recipe baseline despite having the extra runtime. Figure 3 shows that the outcome only stays at zero 0 under contract runtime; however, it is not very clear. It is possible that it reflects the runtime's strict gating rather than the weakness of the outcome reward, since the recipe baseline trains the same task and it works fine with the outcome-only RL. It would be interesting to see if the authors could train recipe with the milestone credit added (no runtime) and compare it against the plain recipe baseline. That would be helpful to answer whether the milestone reward itself contributes anything.
  5. The current pipeline uses only one skill per domain. It would be interesting to see more skills per domain, as these models are capable of handling and selecting skills from a pool.
Questions for authors
  1. Why does the prompt baseline perform so poorly on the Task outcome metric? The performance is very bad even when compared to the direct baseline. I would be curious to learn if the authors found or have any interesting insights about this.
  2. Could the authors also report PCR for the recipe checkpoints evaluated under the runtime? (as was done for the prompt baseline). Search R1-style policy already follows think, search, answer natively. So these experimental results could strengthen the paper's claim because, in the current state, there is no apple-to-apple baseline that could be compared with the proposed method.
  3. Could the authors also evaluate the contract-trained checkpoints in the direct style? That is, no runtime, no feedback, just like the recipe. Table 2B already gives some results, and removing the hints degrades the search domain severely. So studying this behavior on other models or benchmarks would be an important comparison.
  4. It would also be interesting to see some sort of cost analysis between the proposed method and the recipe baseline. I would expect the proposed method to be more expensive in terms of tokens and turns, both during eval and training.
Reviewer_jXp2OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks how to train an agent to follow an expert-written procedure rather than just produce a correct final answer. Outcome rewards do not enforce this, and most skill research does not address strict adherence. The authors turn each skill into a contract: instructions paired with a hand-written state machine that checks actions and returns hints. Training uses GRPO with rewards for first reaching each milestone. Experiments cover math with code and multi-hop search across five small Qwen models, with ablations on one model. Protocol completion is near perfect, but task accuracy is mixed relative to outcome-only training.

Critical strengths
  • The research question of obtaining dense verifiable training signal is useful and important.
  • The motivation makes sense.
  • The authors are honest about limitations and close related works.
  • Give lots of details to reproduce the work.
Critical weaknesses
  • The paper's idea is limited in novelty, with closely matching related work. To me, related works like GLARE (ICML) and HASP (EMNLP) are very similar. I don't quite get the novelty of this paper.
  • The methodology is also only evaluated on math and search-based QA tasks, and is not trivial to generalize to any task (due to the need of creating a procedural skill document). I am not even convinced that many tasks can be defined in this way.
  • The advantage calculation does not fully leverage the dense reward for credit assignment. It is still a trajectory-level advantage.
  • The performance gain is moderate, and not always consistent.
  • It is questionable if the gain is diminishing using some larger models, which can decompose the task itself.
Questions for authors
  • Can you explain the novelty over those close related works more clearly?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.