- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TACO searches for programmatic control policies using two coupled levels. An outer progressive-widening/UCB tree searches structured natural-language policies consisting of a description and staged plan. An inner process holds each plan fixed while refining executable Python code from rollout feedback. An LLM judge classifies realization as partial, full, or abandon, governing refinement termination and subsequent semantic expansion. A shared language knowledge library transfers execution-derived facts across branches. The paper evaluates five MiniHack tasks with the full 121-action NLE space, compares against MLES adapted to UCB search and direct generative code optimization, and analyzes refinement, representation, judging, and knowledge.
Critical strengths
-
The distinction between a strategy and its code realization addresses a real problem in LLM-guided policy search: a useful strategy can be discarded because its first implementation is defective. Committing to multiple execution-guided refinements while holding the NLP fixed is a coherent response. The realization judge also distinguishes following a plan from obtaining a high task return, allowing a faithfully implemented but poor strategy to inform later exploration. This is more substantive than merely attaching a natural-language explanation to each generated program.
-
The empirical study includes relevant baselines, repeated search runs, and two search-cost measures. Figure 2 reports substantial gains on WoD, Quest-Easy, and KeyRoom, with competitive performance on the two corridor tasks. The experiments use 10 independent runs on the first three tasks and 5 on the corridor tasks, and count all generated output tokens, including judge and knowledge outputs. MLES-UCB controls the broad search mechanism, while GCO provides a direct-code refinement comparison. The structured-plan and judge ablations further support the usefulness of the overall design.
Critical weaknesses
-
The reported best discovery score is not an independent estimate of the selected program's expected performance. Figures 2, 4, and 5 use the best score observed during search. Appendix B, Tables 2-4, evaluates each Quest-Easy candidate on only five episodes and candidates elsewhere on 20. No separate reevaluation of the selected programs on fresh environment seeds is specified. Selecting the maximum among many noisy estimates can make both final quality and discovery speed optimistic, and can affect methods differently when their search paths expose different candidates. For example, the score of 1 defined at p5, lines263-264 can mean 5/5 successes on search evaluation episodes; independent reevaluation is needed to support a robust task-solved conclusion. The independent search runs and standard-deviation bands are valuable, but do not remove this selection effect. This leaves a central claimed benefit--the quality of the discovered policies--insufficiently validated.
-
The specification does not consistently establish that a node's retained code realizes the strategy whose value the outer search is estimating. The formulation defines V(psi) over programs realizing psi (p3), but Section 3.3 retains the highest-return program while describing the node verdict as the terminating verdict of the current realization (p5, lines231-246). Refinement may reduce return, so the best retained code and current code can differ. Figure 1 instead says the node inherits its best policy's verdict. Which rule is implemented matters: a full verdict could otherwise cause expansion to preserve stages not realized by the inherited best code. Appendix C.5 treats earlier stages as confirmed during stagewise judging, but it is unclear whether confirmations reset for every new code candidate and all stages are rechecked. Figure 3 measures the same judge's full labels, without an independent fidelity audit. These gaps weaken the interpretation that the method separates strategy quality from implementation quality, even if the discovered executable policies themselves remain useful.
Questions for authors
-
Can the frozen selected programs be reevaluated at final or representative matched-budget checkpoints on a substantial fresh set of environment seeds, including fresh layouts where the task already randomizes them, without further updates and with uncertainty estimates? Please specify whether search evaluations reuse reset seeds and whether test rollouts were ever supplied to the optimizer. Do the three hard-task gains and task-solved conclusions persist? This requests validation of the existing discovered policies, not success on a new benchmark suite.
-
What verdict/routing rule replaces the judge in Figure 5? Since verdicts also select Refine versus Discover expansion and abandonment, the comparison does not isolate adaptive stopping unless this routing is controlled. Are knowledge updates from the judge's new_knowledge field used, and are they held comparable across variants? A fixed-refinement control with terminal judging and matched routing would distinguish these effects.
3 comments
Thank you for the substantial revisions. The held-out evaluation and terminal/adaptive judge controls address my main concerns. Please clarify whether the code/verdict mismatches, reported in 4% of fully implemented nodes, affect successful search paths or final selected programs. The 85% agreement with a stronger LLM provides useful consistency evidence, but should not be interpreted as ground-truth validation of realization fidelity. These limitations warrant qualified mechanism claims. Overall, the expanded empirical evidence supports increasing my recommendation.
Clarification
We thank the reviewer for the positive reassessment.
Code/verdict mismatches. We traced all mismatched nodes through the search trees across all runs. None of these nodes appear on a search path leading to a final selected program. Thus, these mismatches do not affect the final selected programs or their reported performance.
Judge fidelity. We agree that agreement with a stronger LLM provides evidence of consistency, but does not establish ground-truth realization fidelity. We will clarify this limitation in the paper and qualify our claims accordingly.
We hope these clarifications address your remaining concerns. If so, we would appreciate it if you could consider updating your recommendation.
Response
We thank the reviewer for the thorough and detailed comments. Below we address and answer all the questions. We believe addressing them improved the clarity and soundness of our work.
W1/Q1) Held-out test performance.
We appreciate the Reviewer's comments and agree that independent evaluation provides a more reliable measure of the quality of discovered programs. In the revised version, we have updated the evaluation protocol (Section 4). In particular, during search, we evaluate each candidate program on a fixed number of episodes, using a fresh random environment seed for every episode. Search seeds are never reused, and test rollouts are never exposed to the optimizer. Figure 2, 4, 5, and the ones we added during the rebuttal now report the held-out return performance. At each checkpoint, the selected program is evaluated, without further modification, on a fixed set of 100 held-out test seeds disjoint from search seeds and shared across methods, including fresh layouts where applicable. Despite this change, the considerations and conclusions about our proposed method remain unchanged (Appendix H).
W2) Misalignment between retained code and value estimate. We thank the Reviewer for pointing this out! We revised Figure 1 accordingly. In our implementation, a node's verdict is determined by its final realization, whereas its value and the program passed to its children are determined by its best-return realization. This design choice serves two complementary purposes: (i) the value guides the outer search toward promising branches as a practical proxy for V(ψ), (ii) the verdict determines whether expansion should refine the current plan or discover new stages.
A disagreement arises only when an earlier realization outperforms the final realization judged as fully implemented, as ties favor the newer program. This scenario is very rare and only occurs in 4% of fully implemented nodes across all runs and environments. If the judge’s verdict was correct, retaining the higher-return program preserves useful behavior that may not yet be captured by the realized plan, providing an opportunity for further improvement. On the other hand, if the judge incorrectly marks an NLP as “fully implemented,” a later expansion can correct the error with refinements over its stages.
About stagewise judging. At each refinement step t, the judge produces a verdict which includes re-verifying stages inherited from a parent (Section 3.3).
About judge fidelity. To assess the reliability of the realization judge, we independently re-labeled the verdicts underlying Figure 3 using Claude Sonnet 5.5 (Appendix I). Our judge (Gemini-3.1-Flash-Lite) agrees with Claude Sonnet on 85% of verdicts, although our judge assigns full realization somewhat more frequently (Appendix I, Figure 11). Additionally, we re-compute the results of Figure 3 while matching the agreement between our judge and Claude Sonnet (Appendix I, Figure 10). This new analysis yields lower absolute realization rates but preserves the same trend: the proportion of fully realized NLPs increases with additional refinement steps. This supports the importance of allowing multiple refinement steps rather than relying on a single realization attempt.
Q2) Judge ablation (Figure 5).
We thank the Reviewer for the insightful comment, and following the Reviewer’s suggestion, we revised the manuscript and extended our ablation study (Section 3.3, Section 5.2 and Figure 5).
About the judge's new_knowledge field. There are two ways we add information to the knowledge: one way is that at each refinement step, LLMknow will add to it, and the second way is that the judge can also add knowledge during its verification process. We modified Section 3.3 accordingly.
Improved Figure 5. From the submission’s version, we kept the return-based (previously referred as without judge) variant that assumes a fully implemented verdict when the refined program improves over the parent's program and is otherwise partial. This variant never returns the verdict abandon. Additionally, we added two new variants: terminal judge and adaptive judge. The former has a fixed number of refinement steps and only judges the terminal refined program with the three possible verdicts. This way, it will match the expansion routing of TACO. The latter judges at every refinement step, and a fully implemented or abandoned verdict stops the refinement early. All these variants do not modify the knowledge and only the LLMknow will do so, if needed. When comparing these variations with TACO, we can see that the main benefit from the judge is the adaptiveness. We modified Figure 5 to include all these variants.
We hope these clarifications address the reviewer's concerns and we would be happy to answer further questions during the discussion period.