acceptodds
Under review as a conference paper at ICLR 2027

SKILL-AS-CONTROLLER: A NEW PARADIGM FOR SKILL UTILIZATION IN LONG-HORIZON AGENTS

Abstract

Skills are crucial for LLM agents performing long-horizon tasks. However, when used only as prompt context, their effect on action selection remains implicit and suboptimal. Our analysis reveals a skill utilization gap: agents often violate skill rules even when relevant skills are provided, while the same model can often detect these violations afterward. We therefore propose **Skill-as-Controller**, a new paradigm that uses skills to explicitly control the action distribution. At each action step, Skill-as-Controller evaluates candidate actions based on the current state and skill rules, producing skill–action consistency scores. These scores are then combined with the base action distribution through a KL-regularized policy update, yielding a closed-form skill-controlled policy without model parameter updates. Experiments on three long-horizon benchmarks with diverse LLM backbones show consistent improvements in task success and interaction efficiency. On ALFWorld, Skill-as-Controller enables Qwen-3.5-4B to achieve a 92.5% success rate, exceeding the baseline by 58.9 points, while reducing the average number of interaction steps from 24.93 to 12.40. Further analysis shows that Skill-as-Controller reduces common failures, such as state loops and prerequisite violations, while the consistency scores make the effects of skills easier to inspect. These results establish Skill-as-Controller as an effective, efficient, and interpretable paradigm for skill utilization in long-horizon agents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM agents, agent skills, long-horizon tasks, policy control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_F1ZbOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates skill utilization for long‑horizon LLM‑based agents. The authors identify two key empirical observations: (1) the skill‑utilization gap — agents frequently violate skill rules even when relevant skills are provided as prompt context under the common Skill‑as‑Context paradigm; (2) capability asymmetry — the same model can detect some of these violations in post‑hoc evaluation, despite generating violating actions during decision‑making. Motivated by these findings, the authors propose Skill‑as‑Controller: instead of only feeding skills into prompt context, skills are decomposed into evaluation rubrics to compute per‑candidate skill‑action consistency scores. A KL‑regularized closed‑form policy update re‑weights the base agent policy with these consistency scores without fine‑tuning model weights. Evaluations are conducted on three long‑horizon interactive benchmarks (ALFWorld, WebShop, ScienceWorld) across diverse open‑source / closed‑source model backbones. Results show substantial improvements in task success rate and reduction of interaction steps compared to Skill‑as‑Context baselines. Extensive ablations validate component necessity and hyper‑parameter robustness.

Critical strengths
  1. Two novel, empirically‑grounded core observations constitute the main conceptual contribution of this work.
    • First, the paper identifies and quantifies the skill‑utilization gap: under the dominant Skill‑as‑Context paradigm, simply inserting correct relevant skills into prompt context does not guarantee rule‑compliant agent actions. Even with valid skills available, agents frequently produce skill‑violating behaviors. The authors measure per‑step action‑level compliance across three benchmarks, showing only modest compliance improvements brought by in‑context skills, despite widespread adoption of this paradigm in prior agent work. Prior papers mostly reported final task failure but did not isolate this gap as a distinct empirical phenomenon, nor quantify intermediate action‑level compliance under fixed skill content.
    • Second, the paper demonstrates within‑model capability asymmetry via a controlled self‑judgment experiment. Using violation samples generated by the target model itself, the study shows that the same model which outputs violating actions can still detect a meaningful subset of those violations in post‑hoc single‑step evaluation. This reveals an internal capability divide: violation‑recognition capacity exists within the model, yet it is not exercised during action‑sampling. While prior work used episode‑level reflection or external judges, none conducted this within‑model, per‑step controlled experiment to demonstrate this asymmetry.
  2. The observations directly motivate the new Skill‑as‑Controller paradigm. Instead of continuing to rely on implicit prompt‑based guidance, the authors turn the model’s inherent violation‑judgment capacity into explicit per‑candidate‑action scoring signals. Skill rules are converted into evaluation rubrics, and a KL‑regularized closed‑form update modulates the base action distribution at test‑time without fine‑tuning model weights. This provides a concrete inference‑time solution responding exactly to the two uncovered phenomena.
Critical weaknesses

Even after applying Skill‑as‑Controller, substantial residual failure cases remain:

  • ALFWorld best case reaches 92.5% success (7.5% failures remain); WebShop performance stays moderate (38.8% ~ 68.2% across models).
  • Under Skill‑as‑Controller, ALFWorld still has 7.1% state‑loop occurrence and 17.9% prerequisite‑violation rate.

The paper provides high‑level explanations (rubric quality, β trade‑off, indispensability of base‑policy prior), but lacks systematic diagnostic experiments to disentangle root causes of residual failures. Potential contributing factors are hypothesized but not quantified:

  1. Evaluator noise and missed violations: rubric evaluators themselves are LLM‑powered and only achieve 63.2% detection rate on violation samples; evaluator mis‑judgment propagates downstream into policy errors.
  2. Soft‑constraint intrinsic limitation: KL‑reweighting only produces probabilistic bias, not hard filtering. Even given perfect evaluators, violating actions retain non‑zero sampling probability.
  3. Agent state‑perception error: wrong environment observations fed into evaluators lead to incorrect scoring.
  4. Simple averaging aggregation may mishandle conflicting multi‑rule skill requirements; alternative aggregation schemes are not ablated.
  5. Single‑step evaluation limitation: evaluators judge local step‑wise suitability and cannot foresee long‑horizon bad trajectory consequences.
  6. Outcome‑guided evaluator‑set selection on training tasks carries potential over‑fitting risk; generalization gap on held‑out test tasks is unmeasured.
Questions for authors

Q1. Among test‑set failures of Skill‑as‑Controller, what fraction arises because evaluators assign high scores to true violations, versus cases where evaluators score correctly but violating actions are still sampled under soft KL‑regularized policy? Please provide supporting statistics.

Q2. How does the performance of outcome‑selected evaluator sets differ between training tasks and held‑out test tasks? Is there measurable over‑fitting of rubric behavior to training trajectories?

Q3. Please elaborate on failure cases where single‑step evaluators assign locally reasonable scores yet lead to poor long‑horizon trajectories. What is the estimated magnitude of this effect across benchmarks?

5 comments
AuthorsOpenReview ↗

Response to Reviewer F1Zb (1/3)

We thank the reviewer for the detailed review. We added the requested systematic diagnosis: Table R1 attributes every test failure to a cause, and Tables R2 and R3 quantify the remaining factors. On ALFWorld, the main results (Table 2) use the online LLM evaluator, in which the LLM applies each rubric. We also report a compiled evaluator, which executes each rubric as a small program over predicates parsed from the public observation, without any LLM call; it reaches 88.8% on ALFWorld (online: 90.3%).

Residual failures.

Table R1 (under Q1) attributes every test failure to one of four causes. Most residual failures come from the evaluator's judgments and inputs rather than from the soft control: missed violations and missing compliant candidates or misread states account for 66.1% of the 227 failures, and π0\pi_0 overriding a correct evaluation for only 14.5%. We also clarify four points from the review:

  1. Actions are selected greedily (Algorithm 1, line 11), not sampled. A violating action is chosen over a compliant candidate only when the log-odds preference of π0\pi_0 for it exceeds βΔF\beta \Delta F (Appendix B), where ΔF\Delta F is the difference between their consistency scores.
  2. 63.2% is measured on hard cases, and our compiled evaluator detects 69.6%. The 125 violations of Table 1(b) were chosen by the model itself under Skill-as-Context (Table R2). Control also does not need to flag every violation; it only needs to rank some compliant candidate above the violating one.
  3. The 7.1% and 17.9% are shares of trajectories with at least one state loop or prerequisite violation, down from 81.4% and 35.0% under Skill-as-Context (Table 4(a)). They are not failure rates: on the same split, our method solves 90.7% of the tasks.
  4. The 38.8–68.2% on WebShop reflect a strict task. Success requires the product and every requested option to match exactly; for reference, human experts in the original WebShop paper succeed on 59.6% of the test instructions (Yao et al., 2022). With our method, all backbones except Qwen-3.5-4B reach 58.0–68.2%.

Q1: Missed violations versus soft control.

Across the three benchmarks, 30.8% of the failures (70 of 227) arise because the evaluator misses the violation, and only 14.5% (33) because π0\pi_0 overrides a correct evaluation (Table R1). We labeled the violating steps of every test failure with the GPT-5.5 judge of Table 1(a) and assigned each failure to one cause:

  • Missed by the evaluator: the evaluator scores the violating action at least as high as every compliant candidate.
  • Overridden by π0\pi_0: the evaluator scores a compliant candidate higher, but the preference of π0\pi_0 for the violating action outweighs the difference.
  • Compliant steps, failed trajectory: every step is judged compliant, but the trajectory still fails, e.g., because exploration exhausts the step limit.
  • No compliant candidate or misread state: no candidate at the violating step complies with the skill, or the evaluator misreads the current state.

Table R1. Causes of test failures (% of failures), Qwen-3.5-35B-A3B. Added to Appendix F.7 of the revised manuscript.

BenchmarkFailuresMissed by the evaluatorOverridden by π0\pi_0Compliant steps, failed trajectoryNo compliant candidate or misread state
ALFWorld1338.5%30.8%0.0%30.8%
WebShop18531.9%14.6%22.2%31.4%
ScienceWorld2920.7%6.9%10.3%62.1%
All22730.8%14.5%19.4%35.2%

We also tested hard filtering, which keeps only the candidates with the highest consistency score (β → ∞) and chooses among them by π0\pi_0. It removes the failures in which π0\pi_0 overrides a correct evaluation, but lowers success from 88.8% to 81.3% (compiled evaluator, ALFWorld), because it also enforces every misjudgment of the evaluator. The remaining headroom therefore lies mainly in better evaluators and candidate sets rather than in stricter control.

AuthorsOpenReview ↗

Response to Reviewer F1Zb (2/3)

Q3: Limitations of single-step evaluation.

This effect accounts for 19.4% of all failures: none of the 13 on ALFWorld, 22.2% on WebShop, and 10.3% on ScienceWorld. These are the failures with compliant steps but a failed trajectory: every step looks reasonable locally, yet the task fails. All 41 such failures on WebShop end at the step limit without a purchase. Below is an excerpt of one of them (an eco-friendly fleece throw blanket with a given color and size, under 50 dollars); the full trajectory with all candidates and scores is in Appendix F.8 of the revised manuscript.

StepsActionScoreJudge
4 timessearch with the full instruction (same query)+1compliant
open a plausible blanket (a different one each time)+1compliant
back to search: requested color or size not offered+4compliant
Thensearch, then open the fifth blanket, the only one offering both options+1, +1compliant
Lastselect the requested color; the step limit is reached+1compliant

Every step is locally reasonable and scored positively, but each inspection costs three steps, and the product with both options, whose title lists a different size, is reached only at the end. The failure lies in how the steps add up against the budget, which no single-step rule scores. The skills already encode part of the long-horizon structure (e.g., "Only after cleaning, keep holding the target and go to a matching destination receptacle"); evaluators that account for the remaining budget or estimate the value of a plan could address the rest, and we discuss them as future work in the Limitations section (Appendix L of the revised manuscript).

The six factors.

Our analyses address the six factors as follows:

  1. Missed violations: 30.8% of the failures (Table R1); on the 125 violations of Table 1(b), the compiled evaluator detects 69.6% (Table R2), more than LLM judgment with the same rubrics (63.2%), so missed violations can be reduced without changing the rubrics.
  2. Soft constraint: π0\pi_0 overrides a correct evaluation in only 14.5% of the failures; actions are chosen greedily, and hard filtering lowers success (Q1).
  3. State perception: misread states, together with missing compliant candidates, account for 35.2% of the failures and 62.1% on ScienceWorld (Table R1).
  4. Aggregation: applying the rules of all six ALFWorld task families to every task adds many potentially conflicting rules but changes success by only 2.2 points (compiled evaluator), so the simple aggregation seems to handle such conflicts reasonably well; enforcing the rules strictly lowers success (Q1).
  5. Single-step evaluation: failures with compliant steps but a failed trajectory, 19.4%; see Q3.
  6. Over-fitting of the selection: Table R3 (Q2, in our third response).

Table R2. Detection rate on the same 125 violations as in Table 1(b), Qwen-3.5-35B-A3B. Added to Appendix F.9 of the revised manuscript.

JudgmentDetectedRate
Direct judgment with the original skill (Table 1(b))53 / 12542.4%
LLM judgment with the rubrics (Table 1(b))79 / 12563.2%
Compiled evaluator87 / 12569.6%
AuthorsOpenReview ↗

Response to Reviewer F1Zb (3/3)

Q2: Training versus test performance of the selected evaluator sets.

We see no measurable over-fitting. The selected evaluators reach 93.0% (119/128) on the 128 training tasks used for selection, and 90.7% and 90.3% on the two official held-out splits (Table R3): valid_seen (140 tasks) uses the same rooms as the training tasks, and valid_unseen (134 tasks, our test split) uses new rooms. The gap of 2–3 points is within the variation we observe between two runs of the same configuration (about 3 points), and success does not drop further on the new rooms of valid_unseen.

Table R3. ALFWorld success rate (%) on the training tasks used for selection and on the two official held-out splits, Qwen-3.5-35B-A3B. valid_seen shares rooms with the training tasks; valid_unseen is the test split of Table 2. –: not applicable (no selection). Added to Appendix F.10 of the revised manuscript.

MethodTraining tasks used for selection (128)valid_seen (140 tasks)valid_unseen (134 tasks)
Base–40.042.5
Skill-as-Context–51.449.3
Ours (selected evaluators)93.090.790.3

The selection step itself also leaves little room for over-fitting. It only chooses among candidates generated before any task outcome was seen and does not edit them, and its contribution to the test results is small: the unselected sets already improve on Skill-as-Context by 35.5 points, and selection adds only 5.5 more (84.8% → 90.3%).

Presentation.

In the revised manuscript, we have updated Section 3.2, Section 4.1, and the description of Table 3 in Section 4.3, and added a Limitations section (Appendix L) that summarizes the residual failure modes discussed above.

We thank the reviewer again and hope the new analyses address the concerns about residual failures.

Reviewer_bPv7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates the limitations of using natural language skills as prompt context in long-horizon LLM agents and proposes Skill-as-Controller, a framework that explicitly incorporates skill guidance into action selection. The authors identify a skill utilization gap in the conventional Skill-as-Context paradigm, where agents may select actions that violate skill rules even when the relevant skills are provided in the prompt. To address this limitation, the proposed method decomposes a given skill into individual skill rules and constructs an evaluator for each rule. Each evaluator takes the current state and a candidate action as input and assesses the action's consistency with the corresponding rule, producing a skill consistency score for each candidate action. These scores are then used to reweight the base action distribution obtained from the LLM, enabling explicit control over the action distribution. The proposed method is evaluated on three benchmarks, ALFWorld, WebShop, and ScienceWorld, using a diverse set of open-source, closed-source, and fine-tuned models, demonstrating improvements in task success rates over the baselines. Furthermore, the paper analyzes three types of behavioral errors and demonstrates that the proposed method mitigates common failure modes in long-horizon tasks.

Critical strengths

S1. Effective Skill Utilization
The paper empirically demonstrates that the conventional Skill-as-Context approach may fail to ensure sufficient compliance with skill rules, even when the relevant skills are provided in the prompt, and can even degrade performance in certain environments. To address this limitation, the authors propose an approach that explicitly controls the action distribution rather than relying solely on skills as contextual guidance. Experimental results show that the proposed method achieves higher skill rule compliance and task success rates than Skill-as-Context, supporting the effectiveness of explicit skill control.

S2. Comprehensive Experimental Evaluation and Significant Performance Improvements
The proposed method is extensively evaluated across three benchmarks, ALFWorld, WebShop, and ScienceWorld, using a diverse set of open-source, closed-source, and fine-tuned models. The results demonstrate consistent improvements in task success rates over the baselines across different model architectures and parameter scales, with particularly substantial gains for some models.

Critical weaknesses

W1. Lack of Computational Cost Analysis
The proposed method applies KK LLM-based evaluators to each of the ∣At∣|\mathcal{A}_t| candidate actions at every decision step, requiring a total of K×∣At∣K \times |\mathcal{A}_t| rule-action evaluations, which may introduce additional computational overhead compared to Skill-as-Context. Furthermore, generating multiple evaluator sets and selecting the best one by comparing their task success rates also incurs additional computational costs. A quantitative analysis of the overall inference cost and wall-clock time, accounting for these additional computations, would help provide a clearer understanding of the trade-off between performance improvements and computational cost.

W2. Strong Assumptions on Skill Quality and Limited Robustness Analysis
The paper appears to assume that the provided skills are relevant to the current task and contain sufficient information for successful task completion. The experiments also seem to rely on skills that are selected according to task types and manually refined. However, practical skill libraries may contain irrelevant, incomplete, or incorrect skill rules accumulated from diverse tasks. For example, a skill rule acquired from another task might assign a high consistency score to an action that is not appropriate for the current task, potentially affecting action selection. Further analysis under such imperfect skill conditions would help provide a more comprehensive evaluation of the robustness and generalizability of the proposed method.

Questions for authors

Q1. In Table 2, the proposed method appears to achieve higher performance with smaller models than with larger ones in some cases (e.g., Qwen-3.5-4B vs. GPT-OSS-120B on ALFWorld). Is there a particular reason for this observation?

Q2. Could the authors provide a more detailed explanation of the + Step-Inst. and + Act-Score Inst. variants presented in Table 3?

7 comments
AuthorsOpenReview ↗

Response to Reviewer bPv7 (1/3)

We thank the reviewer for the positive assessment. On ALFWorld, the main results (Table 2) use the online LLM evaluator, in which the LLM applies each rubric; we now also report a compiled evaluator that executes the same rubrics without an LLM. Before evaluation, each rubric is written once as a small JSON program: an ordered list of cases, each pairing a condition on the current state with the candidate actions it scores +1 or −1, expressed with a fixed set of predicates (e.g., holding_target, is_transform_target). A rule-based parser computes these predicates from the public observation, action history, and candidate actions (no hidden environment state), and a small interpreter executes the programs, so no LLM is called during evaluation. The compiled evaluator reaches 88.8% on ALFWorld (online LLM evaluator: 90.3%).

W1: Computational cost.

With the compiled evaluator, our method uses about 4 times as many tokens per task as the baselines and takes 1.7–1.8 times as long, while raising success from 42.5–49.3% to 88.8% (Table R1; ALFWorld, Qwen-3.5-35B-A3B; estimates):

  • Rule evaluation (the K×NK \times N evaluations per step): about 13 ms per decision with the compiled evaluator; the online LLM evaluator adds 37.6K tokens per task (36% of its total) and takes 72.4 s per task (about 6.5 s per step).
  • Candidate scoring: the main cost, about 5.3K prompt tokens per step, versus 0.6K–0.8K for generating one action with the baselines; our method needs 12.39 steps instead of 22.43 and 20.44, which offsets only part of this cost.
  • Baselines with similar or more tokens: placing the rubrics in the prompt instead of the skill (64.1K tokens per task) reaches 53.7%, and sampling 8 actions per step for Skill-as-Context and keeping the most probable one (83.1K) reaches 51.5%, versus 88.8% for our method (67.7K).
  • One-time cost: generating one candidate rubric takes about 12.6K tokens per rule, so five candidates for each of the 43 ALFWorld rules take about 2.7M tokens in total. Evaluating the five candidate sets on the 128 training tasks for selection takes about 43M tokens (640 episodes; about 5.6 hours if run sequentially). Both are done once per skill, before evaluation, and together cost about as much as running our method on 680 test tasks; selection, which accounts for most of this cost, can be skipped while keeping about 87% of the gain (84.8% on average over five unselected sets vs. 90.3% for the selected set).

The compiled evaluator is therefore the practical choice for deployment (88.8%, versus 90.3% for the online LLM evaluator). Table R1 is added to Appendix F.2 of the revised manuscript.

Table R1. Average compute per task on ALFWorld, Qwen-3.5-35B-A3B. Token counts and wall-clock times are estimates; wall-clock times were measured separately on idle GPUs. Base and Skill-as-Context generate one action per step (eight in the best-of-8 variant), whereas our method computes the probability of every candidate.

MethodSuccess (%)Avg. stepsTotal tokensEvaluator tokensWall-clock per task (s)
Base42.522.4315,255.4017.1
Skill-as-Context49.320.4417,437.8018.8
Rubric in the prompt instead of the skill53.718.4264,143.5021.6
Skill-as-Context, best of 8 per step51.520.6783,144.0045.2
Ours (online LLM evaluator, Table 2)90.312.61104,632.537,647.372.4
Ours (compiled evaluator)88.812.3967,734.4031.5
AuthorsOpenReview ↗

Response to Reviewer bPv7 (2/3)

W2: Assumptions on skill quality and robustness.

Our method tolerates irrelevant rules and a moderate share of incorrect rules, and it degrades as more rules are incorrect or missing (Table R2). Even with half of the rules incorrect, it stays above Skill-as-Context with correct skills (58.2% vs. 49.3%); with half of the rules removed, it falls to the level of Skill-as-Context (48.5%) and stays above Base (42.5%).

We perturbed the skills on ALFWorld with the compiled evaluator, so that each perturbation applies directly to the rule programs (perturbed rules are chosen with a fixed seed; Appendix J of the revised manuscript):

  • Irrelevant rules: the 43 rules of all six task families are applied to every task, instead of the rules of the task's own family.
  • Incorrect rules: one action condition is flipped between true and false in 20% or 50% of the rules (9 or 22 of 43; only flips that can change a score). For example, the flipped case of take_exact_target in the lamp task no longer penalizes taking a non-target object and instead penalizes the other actions, except the correct take action, which keeps its +1.
  • Incomplete skills: 20% or 50% of the rules of each task family are removed.

Table R2. ALFWorld success rate (%) of our method with perturbed skills, Qwen-3.5-35B-A3B, compiled evaluator. The first row is a rerun of the compiled evaluator in Table R1 (88.8%) and serves as the reference for this experiment; the difference between the two runs reflects run-to-run variation. For comparison, Skill-as-Context reaches 49.3% with the original skills, and Base reaches 42.5%. Added to Appendix F.5 of the revised manuscript.

Skill conditionSuccess (%)Change (points)
Original skills85.8–
Irrelevant: rules of all six task families83.6−2.2
Incorrect: one condition flipped in 20% of the rules79.1−6.7
Incorrect: one condition flipped in 50% of the rules58.2−27.6
Incomplete: 20% of the rules removed64.9−20.9
Incomplete: 50% of the rules removed48.5−37.3

Irrelevant rules cost little because rules whose conditions do not hold return 0 (Eq. 3), and rules for shared subgoals, such as searching for or taking the target, largely agree with the task's own rules. At the same share, removing rules hurts more than corrupting them (64.9% vs. 79.1% at 20%; 48.5% vs. 58.2% at 50%), because a flipped condition matters only where its case is selected, whereas a removed rule gives no guidance at all. Missing rules hurt explicit control more than partly wrong ones; the Limitations section (Appendix L of the revised manuscript) states this and mentions validating the rules on training tasks as future work.

Two clarifications: (i) on ALFWorld, the skill for a task is chosen by its task family, which is identified from the public task description; (ii) the manual simplification only removed task-specific answers and shortcut-like content, without using test outcomes (Section 4.1), and Skill-as-Context and our method use exactly the same skills.

AuthorsOpenReview ↗

Response to Reviewer bPv7 (3/3)

Q1: Why Qwen-3.5-4B outperforms GPT-OSS-120B on ALFWorld.

Yes, we trace it to the base policy π0\pi_0. With the same compiled evaluator for both models, our method still reaches 92.5% with Qwen-3.5-4B as π0\pi_0 and 79.9% with GPT-OSS-120B (Table R3). The reversal occurs only on ALFWorld; on WebShop (38.8% vs. 68.2%) and ScienceWorld (86.9% vs. 90.2%), GPT-OSS-120B is better.

Without control, GPT-OSS-120B is the stronger policy (31.3% vs. 25.4%), but its π0\pi_0 is harder to steer. Under the same compiled evaluator, its π0\pi_0 prefers its top candidate over the second about twice as strongly: the median log-probability gap between the two is 0.50, versus 0.26 for Qwen-3.5-4B (over 3,325 and 1,661 decisions on the 134 test tasks). The control term shifts the log-odds between two candidates by at most 2β2\beta, and β is the same for all backbones, so when π0\pi_0 prefers a violating action by more than 2β2\beta, control cannot overturn it. A larger β closes most of the gap: with the same compiled evaluator, raising β to 10 or 100 lifts GPT-OSS-120B from 79.9% to 90.3% (121/134 in both cases). Both values are large enough for the rule scores to outweigh the preferences of π0\pi_0, so they lead to essentially the same decisions. Stricter control is not better for every backbone: for Qwen-3.5-35B-A3B, strict enforcement (β → ∞) lowers success from 88.8% to 81.3%, so the appropriate β depends on how strongly the base policy prefers its own actions. One possible reason for the stronger preferences of GPT-OSS-120B is that it is a reasoning model: scoring candidate actions without reasoning may make its action probabilities overconfident. We have rewritten the paragraph "Decoupling Competence from Model Scale" in Section 4.2 of the revised manuscript, which overstated this point.

Table R3. ALFWorld success rate (%) of the two models. The last column gives both models the same compiled evaluator. Added to Appendix F.6 of the revised manuscript.

π0\pi_0BaseSkill-as-ContextOurs (Table 2)Ours, same compiled evaluator
Qwen-3.5-4B25.433.692.592.5
GPT-OSS-120B31.351.576.179.9

Q2: The + Step-Inst. and + Act-Score Inst. variants in Table 3.

Table 3 decomposes skill use into three steps: identifying the rules that apply at the current step (Rule Extraction), evaluating candidate actions against them (Action Evaluation), and changing the action distribution accordingly (Policy Control). The two intermediate variants perform the first two steps with the same LLM evaluators as Skill-as-Controller and write their outputs into the prompt, while the action is still generated by the model as in Skill-as-Context:

  • + Step-Inst. keeps the full skill and additionally lists the rules that apply at the current step, i.e., the rules for which the evaluators give a nonzero score to some candidate (rules that do not apply return 0, Eq. 3). Success rises from 49.3% to 58.2%, so identifying the relevant rules helps.
  • + Act-Score Inst. additionally writes the evaluators' score for each candidate action under these rules into the prompt: 72.4%.
  • Skill-as-Controller applies the same scores directly to the action distribution through Eq. 6: 90.3%.

With the same evaluator scores, applying them to the action distribution instead of writing them into the prompt adds another 17.9 points. Both prompts are now in Appendix I of the revised manuscript.

Presentation.

In the revised manuscript, we have updated Section 3.2 (evaluator construction and both implementations), Section 4.1 (evaluation protocols), and the description of Table 3 in Section 4.3.

We hope these results address the reviewer's questions, and we are happy to discuss further.

Reviewer_zWp2OpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes a method to bias an LLM agent's action distribution with action scores from an evaluator that evaluates each action. The evaluator rubric is derived from rules stated in a natural language prompt skill. Comparisons were made against a no-skill baseline and a baseline that has the skill in the prompt. Results show significant performance improvements and decrease in step count.

Critical strengths
  1. Disregarding the fairness concern, the method provides significant performance improvements over baselines.
  2. The method is simple to understand, so its ease of implementation makes it more practical.
Critical weaknesses
  1. My main concern is fairness in the comparison, which is my main reason for recommending rejection at the moment. While the Skill-as-Context baseline takes existing skills and freezes them, the Skill-as-Controller method additionally involves a training phase that tunes action-evaluation rubrics. This risks creating a prompt to the evaluator that is much more informative to the evaluator than the skill is to the Skill-as-Context baseline, and I'm unable to qualitatively assess this as the trained rubrics are not provided (see Q4). I would like to see a controlled comparison where an LLM one-shot generates an action-evaluation rubric with no training, and the Skill-as-Controller is evaluated with that action-evaluation rubric. (In that case, I would also like to see the variance in the result from multiple sampled action-evaluation rubrics.) The simpler approach of just putting the skill in the context of the evaluator is also an ablation that would isolate the effect of the evaluation rubric vs the skill itself. Update: Fully addressed by the rebuttal.
  2. Another major concern is the cost overhead, which would reduce practicality. For each step, the log prob needs to be calculated for every action in the candidate action set, which potentially involves much more inference than the typical rollout which only generates 1 next action at each step. Related to this concern is the fact that the paper makes claims about reduced step count that give the reader the impression of reduced cost, which would be a misleading claim if the cost actually increases. Update: Partially addressed -- the limitation remains, but the paper is no longer misleading regarding the cost.

Update: I've raised my soundness score from 2 to 3 and overall score from 1 to 3.

Questions for authors
  1. How does the skill utilization gap depend on the strength of the model? Section 2 only quantifies the skill utilization gap of the small model Qwen-3.5-35B-A3B. If the gap is primarily found in smaller models that have weaker instruction following capabilities, it would be more honest to add appropriate qualification to statements such as L13 “agents often violate skill rules” / L47 “agents still often select actions that violate its rules”.
  2. How is the action distribution generated? For example, how is the candidate action set obtained, and does it work with infinite action spaces? Is the entire model response regenerated for every action in the candidate action space or are only the tokens that determine the action regenerated? How much does this increase the amount of inference needed?
  3. What does the cost comparison (tokens/dollar amount) look like across the different methods?
  4. What is the prompt template for generating action-evaluation rubrics? What do the trained action-evaluation rubrics look like? If I could read the trained action-evaluation rubrics and compare them with the original skills, that would give me a qualitative understanding of the effect of the training to better assess my fairness concern from W1.
  5. “336 official test variations across eight task families” – how was this subset chosen and is there a particular reason for these eight task families?
  6. (This one is more of a suggestion for future work, but if this is run as a baseline now it would make the paper stronger.) I’m curious how often the rules are programmatically enforceable – if they often are, a natural baseline or alternative approach would be to generate action-evaluation “rubrics” that are programs, and the program can reject an action resulting in a resample (rejection sampling).

Suggestions on writing:

  1. The term “skills” is somewhat overloaded, so spelling out your definition of the term early on would be helpful, e.g., in the abstract and/or in the first paragraph of the introduction. For example, skills in [1] are executable programs, and in RL “skills” refer to option policies.
  2. A derivation of Eq. (6) is likely not necessary, as it is a well-known result. A citation is probably sufficient.

[1] Wang et al. “Voyager: An Open-Ended Embodied Agent with Large Language Models.” TMLR, 2024.

11 comments
AuthorsOpenReview ↗

Response to Reviewer zWp2 (1/5)

We thank the reviewer for the precise suggestions and report all the requested comparisons below.

W1: Fairness of the comparison (main concern).

We ran both requested controls, and the gain holds in both (Table R1):

  • One-shot rubrics without training: 84.8% ± 6.6 over five sampled rubric sets (range 76.1–91.0%), versus 49.3% for Skill-as-Context. The weakest set is still 26.8 points higher, and about 87% of the gain of the selected set (90.3%) remains without selection.
  • Original skill in the evaluator's context, no rubric: 65.7%. This evaluator sees only the raw observation, the action history, the skill text, and the candidate actions, without any predicates or programs. Given the same information as Skill-as-Context, explicit control by itself adds 16.4 points.

Table R1. ALFWorld success rate (%), Qwen-3.5-35B-A3B. Added to Appendix F.1 of the revised manuscript.

SettingSuccess
Skill-as-Context (skill in the prompt)49.3
Rubric in the prompt instead of the skill53.7
Ours, evaluator reads the original skill (no rubric)65.7
Ours, one-shot rubric without selection (mean ± std over 5 sets)84.8 ± 6.6
Ours, selected rubric90.3

The same rubrics give 53.7% when placed in the prompt instead of the skill and 84.8–90.3% when used as a controller, so the gain comes mainly from how the rubrics are used. As evaluators, they make the judgment more accurate (Table 1(b) shows a higher detection rate with rubrics). Table 3 of the paper isolates this further: with exactly the same evaluator scores, writing them into the prompt (+ Act-Score Inst.) gives 72.4%, whereas applying them to the action distribution gives 90.3%.

On ALFWorld, the rubrics are generated once from the skill without seeing any task outcome. Training tasks are used only to choose among rubric candidates generated this way, and the chosen rubrics are not edited. The generation prompt and an example rubric are now in Appendix G of the revised manuscript (see Q4), and all rubric sets will be released with our code.

Evaluator implementation. On ALFWorld, Table 2 uses the online LLM evaluator of Section 3.2, in which the LLM applies each rubric to the candidate actions. We also implemented a compiled evaluator, i.e., the programmatic rubric suggested in Q6, which executes the same rubrics without an LLM. This works because each rubric is written before evaluation as a small JSON program with a fixed schema: given the rule, the whole skill, a fixed list of predicates with their descriptions (23 state and 21 action predicates on ALFWorld), and 10 public states from training tasks without their outcomes, the LLM outputs an ordered list of cases. Each case pairs a condition on the current state (a conjunction of state predicates, e.g., holding_target and transformation_pending) with the candidate actions it scores +1 or −1 (selected by action predicates, e.g., is_transform_target); the first case that applies sets the scores, and all other actions score 0. The predicates are computed by a rule-based parser of the public observation, action history, and candidate actions, written once per benchmark and shared by all skills; it uses no hidden environment state. The predicates do not explain the gain by themselves: the evaluator that reads only the original skill uses none of them and still adds 16.4 points over Skill-as-Context. The compiled evaluator runs the programs with a small interpreter, so no LLM is called at inference time. All 50 rules of ALFWorld and WebShop compile, and the compiled evaluator reaches 88.8% on ALFWorld (online: 90.3%). In the revised manuscript, Section 3.2 now describes both implementations, and Appendix G describes the program format with an example.

AuthorsOpenReview ↗

Response to Reviewer zWp2 (2/5)

Q4: Rubric-generation prompt and generated rubrics.

The generation prompt and the complete rubric of one skill, side by side with its original rules, are now in Appendix G of the revised manuscript; The rubrics restate each rule as checkable conditions and contain no task-specific answers. For example, the rule "At the fridge, cool the held target exactly once" becomes four cases checked in order: the action that cools the target (e.g., "cool lettuce 1 with fridge 1") scores +1 if the agent is at the fridge holding a target that still needs cooling, and −1 if the agent is at the fridge but holds a non-target object or an already cooled target, or if it holds an uncooled target away from the fridge; all other actions score 0.

W2: Computational cost.

We agree: our method needs more inference per task, and the fewer steps do not offset it. By our estimates (Table R2):

  • Tokens: with the compiled evaluator, 67.7K per task, about 4 times as many as Base (15.3K) and Skill-as-Context (17.4K). Almost all of them (66.0K) are prompt tokens for scoring the candidates; generated tokens are similar (1.7K vs. 1.3K).
  • Time: 31.5 s per task, 1.7–1.8 times the baselines (17.1 s and 18.8 s); rule evaluation itself takes about 13 ms per decision. The candidates share the same prompt, so batching them can reduce this time; our implementation does not do this yet.
  • Online LLM evaluator: it adds 37.6K evaluator tokens per task (104.6K in total) and takes 72.4 s per task, so we recommend the compiled evaluator (88.8% vs. 90.3%) for deployment.

Baselines with similar or larger token budgets stay far below our method (88.8%, 67.7K tokens per task): placing the rubrics in the prompt instead of the skill reaches 53.7% (64.1K), and sampling 8 actions per step for Skill-as-Context and keeping the most probable one reaches 51.5% (83.1K). Constructing and selecting the evaluators is a one-time cost per skill (about 46M tokens on ALFWorld, roughly the cost of running our method on 680 test tasks), and selection, which accounts for most of it, can be skipped while keeping about 87% of the gain (W1). With the compiled evaluator, about 4 times the tokens and 13–14 more seconds per task raise success by 39.5–46.3 points. The revised manuscript now uses "interaction-efficient" instead of "efficient" (abstract, Section 1, and Section 4.2), reports compute separately (Appendix F.2), and describes the two action-selection protocols separately in Section 4.1.

Table R2. Success and compute on ALFWorld, Qwen-3.5-35B-A3B. Token counts and wall-clock times are estimated averages per task; wall-clock times were measured separately on idle GPUs. Base and Skill-as-Context generate one action per step (eight in the best-of-8 variant), whereas our method computes the probability of every candidate. Total = agent input + agent output + evaluator input + evaluator output.

MethodSuccess (%)Avg. stepsAgent inputAgent outputEvaluator inputEvaluator outputTotal tokensWall-clock per task (s)
Base42.522.4313,925.61,329.80015,255.417.1
Skill-as-Context49.320.4416,131.91,305.90017,437.818.8
Rubric in the prompt instead of the skill53.718.4262,675.41,468.10064,143.521.6
Skill-as-Context, best of 8 per step51.520.6781,072.82,071.20083,144.045.2
Ours (online LLM evaluator, Table 2)90.312.6165,228.61,756.631,486.16,161.2104,632.572.4
Ours (compiled evaluator)88.812.3966,020.41,714.00067,734.431.5

Q3: Cost comparison (tokens / dollars).

Table R2 gives both the token and the wall-clock comparison (estimates): with the compiled evaluator, our method uses about 4 times the tokens and 1.7–1.8 times the time of the baselines (W2).

AuthorsOpenReview ↗

Response to Reviewer zWp2 (3/5)

Q1: Dependence of the gap on model strength.

The gap narrows for stronger models but remains for frontier models, and we have qualified both statements. For Gemini 3.6 Flash, putting the skill into the prompt raises rule compliance only from 0.597 to 0.612, whereas explicit control raises it to 0.657 (Table R3). At the task level, our method lowers Gemini's failure rate relative to Skill-as-Context from 10.7% to 5.7% on ScienceWorld and from 14.2% to 11.9% on ALFWorld. For GPT-5.5, putting the skill into the prompt leaves ALFWorld success unchanged (85.8% for both Base and Skill-as-Context in Table 2), whereas our method reaches 90.3%. The gap is smaller mainly because stronger models already comply better without the skill (0.597, versus 0.407 for Qwen-3.5-35B-A3B).

Table R3. Action-level rule compliance on ALFWorld, judged by GPT-5.5. Added to Appendix F.3 of the revised manuscript.

BackboneBaseSkill-as-ContextOurs
Qwen-3.5-35B-A3B (Table 1(a))0.4070.4980.554
Gemini 3.6 Flash0.5970.6120.657

Revised statements (abstract and Section 1 of the revised manuscript): L13 now reads "agents, especially smaller models, often violate skill rules even when relevant skills are provided", and L47 "Even when the relevant skill is provided in the prompt, agents still often select actions that violate its rules; this gap is largest for smaller models and narrower, but still present, for frontier models."

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their time and valuable comments!


We have revised the manuscript based on your comments. All changes are highlighted in blue, and the section and appendix numbers in our responses refer to the revised manuscript. The main revisions are:

  • Abstract and Section 1: a definition of skills and qualified statements about the skill utilization gap.
  • Section 3.2: both evaluator implementations: the online LLM evaluator and the compiled evaluator, which executes each rubric as a small program without calling an LLM.
  • Section 4.1: separate descriptions of the action-selection protocols of the baselines and of our method.
  • Appendices F–L (new): the tables of all new results (Appendix F) and a Limitations section (Appendix L).

We are pleased that reviewers recognize the novelty of our findings, the effectiveness and simplicity of our method, and the breadth of our experiments.

1. Novel and empirically grounded findings:

  • “Two novel, empirically‑grounded core observations constitute the main conceptual contribution of this work.” (Reviewer F1Zb)
  • “The paper empirically demonstrates that the conventional Skill-as-Context approach may fail to ensure sufficient compliance with skill rules, even when the relevant skills are provided in the prompt” (Reviewer bPv7)

2. An effective and simple method:

  • “This provides a concrete inference‑time solution responding exactly to the two uncovered phenomena.” (Reviewer F1Zb)
  • “The method is simple to understand, so its ease of implementation makes it more practical.” (Reviewer zWp2)

3. Comprehensive experiments:

  • “The results demonstrate consistent improvements in task success rates over the baselines across different model architectures and parameter scales” (Reviewer bPv7)
  • “Extensive ablations validate component necessity and hyper‑parameter robustness.” (Reviewer F1Zb)

Our answers to the main concerns (details and tables are in our response to each reviewer):

  1. Fairness (zWp2): most of the gain remains without rubric training. One-shot rubrics reach 84.8% ± 6.6 over five sets on ALFWorld (selected: 90.3%; Skill-as-Context: 49.3%). The same rubrics placed in the prompt give only 53.7%, and an evaluator that reads only the original skill still reaches 65.7%.
  2. Compute (zWp2, bPv7): our method costs more, but extra compute alone does not explain the gain. By our estimates, with the compiled evaluator it uses about 4 times the tokens and 1.7–1.8 times the time of the baselines per task, while raising success from 42.5–49.3% to 88.8%. Baselines with similar or larger token budgets reach only 53.7% (rubrics in the prompt) and 51.5% (best of 8 actions per step).
  3. Imperfect skills (bPv7): robust to irrelevant and partly incorrect rules. Irrelevant rules cost 2.2 points, and with half of the rules corrupted, our method (58.2%) still exceeds Skill-as-Context with correct skills (49.3%); missing rules hurt more.
  4. Qwen-3.5-4B vs. GPT-OSS-120B (bPv7): the difference lies in the base policy. With the same compiled evaluator, the 4B model still leads (92.5% vs. 79.9%). The π0\pi_0 of GPT-OSS-120B prefers its top candidate about twice as strongly (median log-probability gap: 0.50 vs. 0.26), and a larger β (10 or 100) lifts it to 90.3%.
  5. Residual failures (F1Zb): mostly due to the evaluator, not to soft control. The base policy overriding a correct evaluation explains only 14.5% of the test failures, whereas missed violations and missing candidates or misread states explain 66.1%; hard filtering lowers success (81.3% vs. 88.8%).
  6. ScienceWorld task families (zWp2): the gain is not specific to the eight families. They are the families covered by our two ScienceWorld skills. On the full official test set (30 families, 1,819 variations), our method improves on Skill-as-Context by about 25 points with both Qwen-3.5-35B-A3B (81.9% vs. 57.1%) and Qwen-3.5-4B (74.2% vs. 49.5%).
AuthorsOpenReview ↗

Summary for the Area Chair

Dear Area Chair,

We thank you and the reviewers for the constructive discussion. Below we briefly summarize our contributions, the main additions from the rebuttal, and the current status of the reviews.

Contributions.

  • Skill utilization gap. Under the common Skill-as-Context paradigm, agents often violate skill rules even when the relevant skill is provided, although the same model can recognize many of these violations when evaluating actions separately. Reviewer F1Zb highlighted these observations as the main conceptual contribution.
  • Skill-as-Controller. We turn skill rules into evaluators of candidate actions and use their scores to control the action distribution through a closed-form KL-regularized update, without updating model parameters.
  • Consistent gains across three long-horizon benchmarks and diverse backbones, as noted by Reviewer bPv7 (e.g., raising ALFWorld success from 49.3% to 90.3% with Qwen-3.5-35B-A3B).

Main additions from the rebuttal.

  • Fairness (zWp2). Most of the gain is retained without any rubric selection (84.8% ± 6.6 vs. 49.3% for Skill-as-Context), whereas placing the same rubrics in the prompt helps much less (53.7%).
  • Compute (zWp2, bPv7). Our method requires more computation than the standard protocol, which the revised paper now states explicitly. A compiled evaluator, which relies on a parser written once per benchmark, removes LLM calls from rule evaluation (88.8% vs. 90.3% for the online evaluator). Baselines with comparable or larger budgets remain well below our method, including scoring the full action set under Skill-as-Context with about twice our tokens (56.0% vs. 88.8%).
  • Generality (zWp2). The gains carry over to the full official ScienceWorld test set (about 25 points across 30 task families), and the skill utilization gap persists, though narrower, for frontier models.
  • Diagnosis (F1Zb, bPv7). Residual failures stem mainly from the evaluator and its inputs rather than from soft control (14.5%), evaluator selection shows no measurable over-fitting, and the method tolerates irrelevant and partly incorrect rules.

Revisions (in blue). We added a definition of skills and qualified our claims about the gap (Abstract, Section 1), described both evaluator implementations (Section 3.2) and the action-selection protocols (Section 4.1), stated the higher per-task compute in the main text (Section 4.2) and the Limitations (Appendix L), and added appendices with the new experiments and analyses.

Status of the reviews. All three reviewers now recommend weak acceptance.

  • Reviewer zWp2 raised the score from 1 to 3, stating that the primary concern is fully addressed and that the remaining concerns, the increased cost and the reliance on quality skills, are minor and now transparently described in the paper.
  • Reviewer bPv7 noted that the new experiments resolved their remaining concerns and strengthened the empirical evidence, and raised the soundness score from 2 to 3.
  • Reviewer F1Zb rated the paper 3; we provided the systematic diagnosis of residual failures requested in the review.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.