- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates skill utilization for long‑horizon LLM‑based agents. The authors identify two key empirical observations: (1) the skill‑utilization gap — agents frequently violate skill rules even when relevant skills are provided as prompt context under the common Skill‑as‑Context paradigm; (2) capability asymmetry — the same model can detect some of these violations in post‑hoc evaluation, despite generating violating actions during decision‑making. Motivated by these findings, the authors propose Skill‑as‑Controller: instead of only feeding skills into prompt context, skills are decomposed into evaluation rubrics to compute per‑candidate skill‑action consistency scores. A KL‑regularized closed‑form policy update re‑weights the base agent policy with these consistency scores without fine‑tuning model weights. Evaluations are conducted on three long‑horizon interactive benchmarks (ALFWorld, WebShop, ScienceWorld) across diverse open‑source / closed‑source model backbones. Results show substantial improvements in task success rate and reduction of interaction steps compared to Skill‑as‑Context baselines. Extensive ablations validate component necessity and hyper‑parameter robustness.
Critical strengths
- Two novel, empirically‑grounded core observations constitute the main conceptual contribution of this work.
- First, the paper identifies and quantifies the skill‑utilization gap: under the dominant Skill‑as‑Context paradigm, simply inserting correct relevant skills into prompt context does not guarantee rule‑compliant agent actions. Even with valid skills available, agents frequently produce skill‑violating behaviors. The authors measure per‑step action‑level compliance across three benchmarks, showing only modest compliance improvements brought by in‑context skills, despite widespread adoption of this paradigm in prior agent work. Prior papers mostly reported final task failure but did not isolate this gap as a distinct empirical phenomenon, nor quantify intermediate action‑level compliance under fixed skill content.
- Second, the paper demonstrates within‑model capability asymmetry via a controlled self‑judgment experiment. Using violation samples generated by the target model itself, the study shows that the same model which outputs violating actions can still detect a meaningful subset of those violations in post‑hoc single‑step evaluation. This reveals an internal capability divide: violation‑recognition capacity exists within the model, yet it is not exercised during action‑sampling. While prior work used episode‑level reflection or external judges, none conducted this within‑model, per‑step controlled experiment to demonstrate this asymmetry.
- The observations directly motivate the new Skill‑as‑Controller paradigm. Instead of continuing to rely on implicit prompt‑based guidance, the authors turn the model’s inherent violation‑judgment capacity into explicit per‑candidate‑action scoring signals. Skill rules are converted into evaluation rubrics, and a KL‑regularized closed‑form update modulates the base action distribution at test‑time without fine‑tuning model weights. This provides a concrete inference‑time solution responding exactly to the two uncovered phenomena.
Critical weaknesses
Even after applying Skill‑as‑Controller, substantial residual failure cases remain:
- ALFWorld best case reaches 92.5% success (7.5% failures remain); WebShop performance stays moderate (38.8% ~ 68.2% across models).
- Under Skill‑as‑Controller, ALFWorld still has 7.1% state‑loop occurrence and 17.9% prerequisite‑violation rate.
The paper provides high‑level explanations (rubric quality, β trade‑off, indispensability of base‑policy prior), but lacks systematic diagnostic experiments to disentangle root causes of residual failures. Potential contributing factors are hypothesized but not quantified:
- Evaluator noise and missed violations: rubric evaluators themselves are LLM‑powered and only achieve 63.2% detection rate on violation samples; evaluator mis‑judgment propagates downstream into policy errors.
- Soft‑constraint intrinsic limitation: KL‑reweighting only produces probabilistic bias, not hard filtering. Even given perfect evaluators, violating actions retain non‑zero sampling probability.
- Agent state‑perception error: wrong environment observations fed into evaluators lead to incorrect scoring.
- Simple averaging aggregation may mishandle conflicting multi‑rule skill requirements; alternative aggregation schemes are not ablated.
- Single‑step evaluation limitation: evaluators judge local step‑wise suitability and cannot foresee long‑horizon bad trajectory consequences.
- Outcome‑guided evaluator‑set selection on training tasks carries potential over‑fitting risk; generalization gap on held‑out test tasks is unmeasured.
Questions for authors
Q1. Among test‑set failures of Skill‑as‑Controller, what fraction arises because evaluators assign high scores to true violations, versus cases where evaluators score correctly but violating actions are still sampled under soft KL‑regularized policy? Please provide supporting statistics.
Q2. How does the performance of outcome‑selected evaluator sets differ between training tasks and held‑out test tasks? Is there measurable over‑fitting of rubric behavior to training trajectories?
Q3. Please elaborate on failure cases where single‑step evaluators assign locally reasonable scores yet lead to poor long‑horizon trajectories. What is the estimated magnitude of this effect across benchmarks?
5 comments
Response to Reviewer F1Zb (2/3)
Q3: Limitations of single-step evaluation.
This effect accounts for 19.4% of all failures: none of the 13 on ALFWorld, 22.2% on WebShop, and 10.3% on ScienceWorld. These are the failures with compliant steps but a failed trajectory: every step looks reasonable locally, yet the task fails. All 41 such failures on WebShop end at the step limit without a purchase. Below is an excerpt of one of them (an eco-friendly fleece throw blanket with a given color and size, under 50 dollars); the full trajectory with all candidates and scores is in Appendix F.8 of the revised manuscript.
| Steps | Action | Score | Judge |
|---|---|---|---|
| 4 times | search with the full instruction (same query) | +1 | compliant |
| open a plausible blanket (a different one each time) | +1 | compliant | |
| back to search: requested color or size not offered | +4 | compliant | |
| Then | search, then open the fifth blanket, the only one offering both options | +1, +1 | compliant |
| Last | select the requested color; the step limit is reached | +1 | compliant |
Every step is locally reasonable and scored positively, but each inspection costs three steps, and the product with both options, whose title lists a different size, is reached only at the end. The failure lies in how the steps add up against the budget, which no single-step rule scores. The skills already encode part of the long-horizon structure (e.g., "Only after cleaning, keep holding the target and go to a matching destination receptacle"); evaluators that account for the remaining budget or estimate the value of a plan could address the rest, and we discuss them as future work in the Limitations section (Appendix L of the revised manuscript).
The six factors.
Our analyses address the six factors as follows:
- Missed violations: 30.8% of the failures (Table R1); on the 125 violations of Table 1(b), the compiled evaluator detects 69.6% (Table R2), more than LLM judgment with the same rubrics (63.2%), so missed violations can be reduced without changing the rubrics.
- Soft constraint: overrides a correct evaluation in only 14.5% of the failures; actions are chosen greedily, and hard filtering lowers success (Q1).
- State perception: misread states, together with missing compliant candidates, account for 35.2% of the failures and 62.1% on ScienceWorld (Table R1).
- Aggregation: applying the rules of all six ALFWorld task families to every task adds many potentially conflicting rules but changes success by only 2.2 points (compiled evaluator), so the simple aggregation seems to handle such conflicts reasonably well; enforcing the rules strictly lowers success (Q1).
- Single-step evaluation: failures with compliant steps but a failed trajectory, 19.4%; see Q3.
- Over-fitting of the selection: Table R3 (Q2, in our third response).
Table R2. Detection rate on the same 125 violations as in Table 1(b), Qwen-3.5-35B-A3B. Added to Appendix F.9 of the revised manuscript.
| Judgment | Detected | Rate |
|---|---|---|
| Direct judgment with the original skill (Table 1(b)) | 53 / 125 | 42.4% |
| LLM judgment with the rubrics (Table 1(b)) | 79 / 125 | 63.2% |
| Compiled evaluator | 87 / 125 | 69.6% |
Response to Reviewer F1Zb (3/3)
Q2: Training versus test performance of the selected evaluator sets.
We see no measurable over-fitting. The selected evaluators reach 93.0% (119/128) on the 128 training tasks used for selection, and 90.7% and 90.3% on the two official held-out splits (Table R3): valid_seen (140 tasks) uses the same rooms as the training tasks, and valid_unseen (134 tasks, our test split) uses new rooms. The gap of 2–3 points is within the variation we observe between two runs of the same configuration (about 3 points), and success does not drop further on the new rooms of valid_unseen.
Table R3. ALFWorld success rate (%) on the training tasks used for selection and on the two official held-out splits, Qwen-3.5-35B-A3B. valid_seen shares rooms with the training tasks; valid_unseen is the test split of Table 2. –: not applicable (no selection). Added to Appendix F.10 of the revised manuscript.
| Method | Training tasks used for selection (128) | valid_seen (140 tasks) | valid_unseen (134 tasks) |
|---|---|---|---|
| Base | – | 40.0 | 42.5 |
| Skill-as-Context | – | 51.4 | 49.3 |
| Ours (selected evaluators) | 93.0 | 90.7 | 90.3 |
The selection step itself also leaves little room for over-fitting. It only chooses among candidates generated before any task outcome was seen and does not edit them, and its contribution to the test results is small: the unselected sets already improve on Skill-as-Context by 35.5 points, and selection adds only 5.5 more (84.8% → 90.3%).
Presentation.
In the revised manuscript, we have updated Section 3.2, Section 4.1, and the description of Table 3 in Section 4.3, and added a Limitations section (Appendix L) that summarizes the residual failure modes discussed above.
We thank the reviewer again and hope the new analyses address the concerns about residual failures.
Response to Reviewer F1Zb (1/3)
We thank the reviewer for the detailed review. We added the requested systematic diagnosis: Table R1 attributes every test failure to a cause, and Tables R2 and R3 quantify the remaining factors. On ALFWorld, the main results (Table 2) use the online LLM evaluator, in which the LLM applies each rubric. We also report a compiled evaluator, which executes each rubric as a small program over predicates parsed from the public observation, without any LLM call; it reaches 88.8% on ALFWorld (online: 90.3%).
Residual failures.
Table R1 (under Q1) attributes every test failure to one of four causes. Most residual failures come from the evaluator's judgments and inputs rather than from the soft control: missed violations and missing compliant candidates or misread states account for 66.1% of the 227 failures, and π0 overriding a correct evaluation for only 14.5%. We also clarify four points from the review:
Q1: Missed violations versus soft control.
Across the three benchmarks, 30.8% of the failures (70 of 227) arise because the evaluator misses the violation, and only 14.5% (33) because π0 overrides a correct evaluation (Table R1). We labeled the violating steps of every test failure with the GPT-5.5 judge of Table 1(a) and assigned each failure to one cause:
Table R1. Causes of test failures (% of failures), Qwen-3.5-35B-A3B. Added to Appendix F.7 of the revised manuscript.
We also tested hard filtering, which keeps only the candidates with the highest consistency score (β → ∞) and chooses among them by π0. It removes the failures in which π0 overrides a correct evaluation, but lowers success from 88.8% to 81.3% (compiled evaluator, ALFWorld), because it also enforces every misjudgment of the evaluator. The remaining headroom therefore lies mainly in better evaluators and candidate sets rather than in stricter control.