- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
To avoid the decision-time latency of online LLM inference or world-model planning, this paper proposes Code to Control, where an LLM directly writes standalone Python reactive controller scripts. In discrete games (Atari and Flappy Bird), the LLM iteratively edits if-else rules over object coordinates extracted by OCAtari. In continuous control (MuJoCo), the LLM open-loop writes elementary math features from random transitions, and Augmented Random Search (ARS, Mania et al., 2018) fits a linear weight matrix . Essentially, the method trades control capability for fast execution speed ( per step) by using simple rule-based scripts.
Critical strengths
- Very low execution latency: Running a plain Python script at test time takes only per step, avoiding online LLM calls or planning rollouts.
- Good sample efficiency on simple games: When ground-truth object bounding boxes and velocities are explicitly given via OCAtari, the LLM can piece together working rule-based scripts (such as linear velocity extrapolation in Pong) within episodes.
Critical weaknesses
-
Focusing purely on speed leads to an outdated paradigm with a very low capability ceiling: The paper's main argument essentially focuses on how fast a step runs, while ignoring whether the controller actually learns meaningful skills. With modern decision models and end-to-end policies advancing rapidly, going back to having an LLM write a few dozen lines of
if-elserules or simple math formulas feels clumsy and outdated. Once tested on harder locomotion tasks (Humanoid, Hopper, Walker2d), this brittle representation hits a hard wall. As reported in Section 3.4, the robots barely move forward over steps (Humanoid displacement is , Hopper is , and Walker2d is ). They just learn to stand still without falling to farm the per-step alive bonus. This directly contradicts the abstract's claim that the method "scales to complex locomotion tasks." -
Missing the most obvious baseline in MuJoCo (vanilla linear ARS, Mania et al., 2018): The continuous control section directly uses ARS (Mania et al., 2018) with online mean-variance normalization to fit . However, Mania et al. (2018) already showed that fitting a linear policy directly on normalized raw MuJoCo states (without any LLM features) achieves strong returns on these exact tasks. Looking at Appendix Box 5, the LLM-written features for HalfCheetah are mostly just raw joint angles and velocities wrapped in
tanh, plus two clock terms. The authors only compare against weak code world-model baselines (GIF-MCTS, WorldCoder), while completely leaving out vanilla linear ARS on raw states. They also do not report how many environment steps ARS actually used. Without this baseline, it is impossible to tell if the LLM features add any value over the ARS optimizer itself. -
Disconnected methodology and missing experimental details: The paper claims a unified framework that separates structure synthesis from parameter fitting, but the discrete and continuous setups are completely different. In Atari, there is no parameter optimizer at all; the LLM blindly edits both code and numbers, and of revisions actually hurt performance (relying on saving the best historical checkpoint). In MuJoCo, there is no closed-loop LLM revision at all; the LLM blindly guesses feature functions from random steps without ever seeing return feedback. Finally, Section 3.3 never explains how numerical constants are automatically extracted and refitted from arbitrary Python scripts containing over a dozen hardcoded numbers (Appendix Box 4), or how many environment steps that refitting takes.
Questions for authors
- Under the exact same ARS hyperparameters and environment-step budget, what are the median returns of vanilla linear ARS on raw observations (Mania et al., 2018) and raw observations with clock terms across the MuJoCo tasks? Also, how many environment steps and episodes did ARS use in your MuJoCo runs?
- Since Humanoid (), Hopper (), and Walker2d () barely move forward and mostly just stand in place to collect alive bonuses, how do you justify the claim in the abstract that the method "scales to complex locomotion tasks"?
- In the Flappy Bird adaptation experiment (Section 3.3), how does your system automatically parse, select, and optimize the hardcoded numbers inside an arbitrary Python script (like Box 4), and how many environment steps are used for refitting? Also, is the PPO baseline in Atari trained on the same OCAtari object coordinates or on raw pixels?
2 comments
Official Comment by Authors (2 of 2)
W3. Disconnected methodology.
We revised the method so that every domain follows one connected loop (Section 2, Algorithm 1). The LLM writes the program's structure, an optimizer fits its numeric parameters from return, and when a change in the environment breaks the structure, the LLM revises it. Atari now fits its constants. Bayesian optimization refits the best program's constants without any LLM call. Section 2.2 and Appendix A.1.2 describe the procedure (Algorithm 2), and Q3 below summarizes it. With 100k extra steps the refit raises Fishing Derby from 27.7 to 43.2 and Space Invaders from 445 to 562 (Table 5).
On revisions that hurt performance. The method keeps the best controller found so far and revises from it, and a revision that scores lower is discarded (Section 2.2, Algorithm 1). A worse revision therefore never replaces the best controller.
Closed-loop structural repair. Some changes in the environment break the program's structure, not only its parameters. Examples are a reversed or swapped actuator, a flipped sensor, or reversed gravity in Flappy Bird. Refitting cannot undo such a change. The LLM therefore revises the structure from the controller's episodes and return. The edit is kept only if the return does not drop, and the parameters are then refitted (Appendix A.1.3). In MuJoCo this restores the pre-fault score in 36 of 36 runs across six tasks, where refitting alone does not. Under reversed gravity in Flappy Bird, LLM edits repair the controller in 8 of 15 attempts (Section 3.3).
Actuator reversed
| Method | Hopper | Walker2d | HalfCheetah | Ant |
|---|---|---|---|---|
| LLM edit | 3,638 | 4,650 | 5,301 | 4,714 |
| ARS refit | 580 | 2,071 | 2,504 | 3,233 |
| Code to Control (LLM edit + ARS) | 3,638 | 4,757 | 5,342 | 4,746 |
Actuator reversed and two swapped
| Method | Hopper | Walker2d | HalfCheetah | Ant |
|---|---|---|---|---|
| LLM edit | 3,639 | 4,656 | 5,301 | 4,714 |
| ARS refit | 652 | 1,524 | 3,415 | 546 |
| Code to Control (LLM edit + ARS) | 3,656 | 4,788 | 5,317 | 4,698 |
Sensor flipped
| Method | Hopper | Walker2d | HalfCheetah | Ant |
|---|---|---|---|---|
| LLM edit | 3,639 | 4,656 | 5,301 | 200 |
| ARS refit | 3,204 | 1,807 | 3,902 | 4,338 |
| Code to Control (LLM edit + ARS) | 3,656 | 4,797 | 5,355 | 4,464 |
MuJoCo return after structural faults, mean of three seeds (Figure 4C). ARS refit uses 25% more environment steps than the original fit. Code to Control refits after LLM edits within the same budget.
Q3. Parsing and refitting constants, and PPO's observation.
The numeric literals of the program are found by parsing its syntax tree. Each nonzero literal is searched on a log scale within [θ₀/4, 4θ₀] around the LLM's value, and a zero literal within [−1, 1]. Bayesian optimization plays one episode per candidate and selects the candidate with the highest predicted score. In Atari, the new constants are kept only if they also win on held-out validation episodes (Appendix A.1.2, Algorithm 2). In Flappy Bird, each refit uses at most 100k environment steps (Section 3.3). The Atari games use the same procedure, with a refit budget equal to the game's step budget minus the steps used by synthesis, for example 87k steps on Fishing Derby. PPO is trained on the same OCAtari object coordinates as our controllers, not on raw pixels (Section 3.2, Appendix A.2).
References
[1] Mania, H., Guy, A., & Recht, B. (2018). Simple random search of static linear policies is competitive for reinforcement learning. Advances in Neural Information Processing Systems, 31.
[2] Bosio et al., Combining Large Language Models and Gradient-Free Optimization for Automatic Control Policy Synthesis. American Control Conference, 2026.
Official Comment by Authors (1 of 2)
We thank the reviewer for a detailed and technical review. Following the reviewer's suggestions, we added vanilla linear ARS with and without clock terms, refit constants in Atari, and added closed-loop structural repair. These changes have strengthened the paper.
W1 / Q2. Capability, and the locomotion claim.
Forward distance in one 1,000-step episode, median over three runs (Appendix A.3.3).
The revised MuJoCo experiments fit at larger budgets (Table 2), and the alive bonus is removed from the fitting objective, so a controller cannot score by standing still. All Code to Control locomotion controllers now walk, and they move further than raw ARS on all six tasks. The program's structure also lets the controller adapt. After an actuator is reversed or two are swapped, closed-loop structural repair restores the pre-fault score in 36 of 36 runs across six tasks, where ARS with 25% more environment steps does not recover (table under W3, Appendix A.3.4). On end-to-end policies, we compare with PPO on Atari and with linear ARS on MuJoCo, which is competitive with deep RL on these tasks [1], and we added Bosio et al. [2], an LLM-based program-as-policy method suggested by Reviewer ooir. Code to Control scores higher than all of them on every Atari game and every locomotion task (Figure 2A, Table 1).
W2 / Q1. Vanilla linear ARS and clock terms.
Test return, median over three runs (Table 1, Table 6).
We added vanilla linear ARS on raw observations, and on raw observations with the clock terms of our feature maps. Both use exactly the same ARS hyperparameters and environment steps as our method. Our controllers have a higher median return than both on all six locomotion tasks, most of all on Humanoid.
Environment steps and episodes. Table 1 and Table 2 give the budget for every task. Humanoid uses 150M steps, Ant 30M, Walker2d and HalfCheetah 20M, Hopper 10M and Swimmer 5M. The other four tasks use 1,440 episodes, at most 1.45M steps.