acceptodds
Under review as a conference paper at ICLR 2027

Code to Control: Synthesizing Parameterized Reactive Controllers

Abstract

Recent LLM-based approaches to control either invoke a language model to select actions or synthesize world models that require planning at every decision, introducing latency that can limit real-time use. We introduce Code to Control, an approach that synthesizes Python controllers which execute directly as policies. Code to Control separates program structure from parameters. An LLM synthesizes the controller structure, while derivative-free search fits its parameters for continuous control using feedback from the environment. Once learned, the resulting controllers require neither LLM inference nor planning at decision time, enabling real-time gameplay and, under our timing protocol, faster action selection than a PPO policy. Across a suite of Atari games, Flappy Bird, and MuJoCo tasks, Code to Control outperforms planning-based program synthesis methods, remains competitive with deep reinforcement learning while using fewer environment interactions, transfers across substantial changes in environment dynamics, and scales to complex locomotion tasks.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
program synthesis, neurosymbolic systems, programmatic policies

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_9QTPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

To avoid the decision-time latency of online LLM inference or world-model planning, this paper proposes Code to Control, where an LLM directly writes standalone Python reactive controller scripts. In discrete games (Atari and Flappy Bird), the LLM iteratively edits if-else rules over object coordinates extracted by OCAtari. In continuous control (MuJoCo), the LLM open-loop writes 1414 elementary math features ϕ(o,t)\phi(o,t) from 4040 random transitions, and Augmented Random Search (ARS, Mania et al., 2018) fits a linear weight matrix WW. Essentially, the method trades control capability for fast execution speed (11.4 μs11.4\,\mu\text{s} per step) by using simple rule-based scripts.

Critical strengths
  1. Very low execution latency: Running a plain Python script at test time takes only 11.4 μs11.4\,\mu\text{s} per step, avoiding online LLM calls or planning rollouts.
  2. Good sample efficiency on simple games: When ground-truth object bounding boxes and velocities are explicitly given via OCAtari, the LLM can piece together working rule-based scripts (such as linear velocity extrapolation in Pong) within 2020 episodes.
Critical weaknesses
  1. Focusing purely on speed leads to an outdated paradigm with a very low capability ceiling: The paper's main argument essentially focuses on how fast a step runs, while ignoring whether the controller actually learns meaningful skills. With modern decision models and end-to-end policies advancing rapidly, going back to having an LLM write a few dozen lines of if-else rules or 1414 simple math formulas feels clumsy and outdated. Once tested on harder locomotion tasks (Humanoid, Hopper, Walker2d), this brittle representation hits a hard wall. As reported in Section 3.4, the robots barely move forward over 1,0001,000 steps (Humanoid displacement is −0.44m-0.44\text{m}, Hopper is 1.46m1.46\text{m}, and Walker2d is 3.12m3.12\text{m}). They just learn to stand still without falling to farm the per-step alive bonus. This directly contradicts the abstract's claim that the method "scales to complex locomotion tasks."

  2. Missing the most obvious baseline in MuJoCo (vanilla linear ARS, Mania et al., 2018): The continuous control section directly uses ARS (Mania et al., 2018) with online mean-variance normalization to fit WW. However, Mania et al. (2018) already showed that fitting a linear policy π(o)=Wo^\pi(o) = W\hat{o} directly on normalized raw MuJoCo states (without any LLM features) achieves strong returns on these exact tasks. Looking at Appendix Box 5, the 1414 LLM-written features for HalfCheetah are mostly just raw joint angles and velocities wrapped in tanh, plus two sin⁡/cos⁡\sin/\cos clock terms. The authors only compare against weak code world-model baselines (GIF-MCTS, WorldCoder), while completely leaving out vanilla linear ARS on raw states. They also do not report how many environment steps ARS actually used. Without this baseline, it is impossible to tell if the LLM features add any value over the ARS optimizer itself.

  3. Disconnected methodology and missing experimental details: The paper claims a unified framework that separates structure synthesis from parameter fitting, but the discrete and continuous setups are completely different. In Atari, there is no parameter optimizer at all; the LLM blindly edits both code and numbers, and 72%72\% of revisions actually hurt performance (relying on saving the best historical checkpoint). In MuJoCo, there is no closed-loop LLM revision at all; the LLM blindly guesses L=6L=6 feature functions from 4040 random steps without ever seeing return feedback. Finally, Section 3.3 never explains how numerical constants are automatically extracted and refitted from arbitrary Python scripts containing over a dozen hardcoded numbers (Appendix Box 4), or how many environment steps that refitting takes.

Questions for authors
  1. Under the exact same ARS hyperparameters and environment-step budget, what are the median returns of vanilla linear ARS on raw observations ϕ(o)=o\phi(o) = o (Mania et al., 2018) and raw observations with clock terms ϕ(o,t)=[o,sin⁡(2πt/20),cos⁡(2πt/13)]\phi(o,t) = [o, \sin(2\pi t/20), \cos(2\pi t/13)] across the 1010 MuJoCo tasks? Also, how many environment steps and episodes did ARS use in your MuJoCo runs?
  2. Since Humanoid (−0.44m-0.44\text{m}), Hopper (1.46m1.46\text{m}), and Walker2d (3.12m3.12\text{m}) barely move forward and mostly just stand in place to collect alive bonuses, how do you justify the claim in the abstract that the method "scales to complex locomotion tasks"?
  3. In the Flappy Bird adaptation experiment (Section 3.3), how does your system automatically parse, select, and optimize the hardcoded numbers inside an arbitrary Python script (like Box 4), and how many environment steps are used for refitting? Also, is the PPO baseline in Atari trained on the same OCAtari object coordinates or on raw pixels?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1 of 2)

We thank the reviewer for a detailed and technical review. Following the reviewer's suggestions, we added vanilla linear ARS with and without clock terms, refit constants in Atari, and added closed-loop structural repair. These changes have strengthened the paper.

W1 / Q2. Capability, and the locomotion claim.

TaskCode to Control (m)Raw ARS (m)
Humanoid38.426.4
Walker2d31.118.2
HalfCheetah307.3283.2
Ant215.0207.8
Hopper20.820.6
Swimmer14.512.4

Forward distance in one 1,000-step episode, median over three runs (Appendix A.3.3).

The revised MuJoCo experiments fit at larger budgets (Table 2), and the alive bonus is removed from the fitting objective, so a controller cannot score by standing still. All Code to Control locomotion controllers now walk, and they move further than raw ARS on all six tasks. The program's structure also lets the controller adapt. After an actuator is reversed or two are swapped, closed-loop structural repair restores the pre-fault score in 36 of 36 runs across six tasks, where ARS with 25% more environment steps does not recover (table under W3, Appendix A.3.4). On end-to-end policies, we compare with PPO on Atari and with linear ARS on MuJoCo, which is competitive with deep RL on these tasks [1], and we added Bosio et al. [2], an LLM-based program-as-policy method suggested by Reviewer ooir. Code to Control scores higher than all of them on every Atari game and every locomotion task (Figure 2A, Table 1).

W2 / Q1. Vanilla linear ARS and clock terms.

TaskCode to ControlRaw ARSRaw + clock
Humanoid8,0136,9381,233
Walker2d4,8803,2764,167
HalfCheetah5,7555,1865,185
Ant4,7574,7044,711
Hopper3,5943,5683,430
Swimmer361311279
HumanoidStandup127,370132,585139,160
Inverted Pendulum1,0001,0001,000
Inv. Double Pendulum9,358.99,359.7129.5
Reacher−9.24−7.01−9.4

Test return, median over three runs (Table 1, Table 6).

We added vanilla linear ARS on raw observations, and on raw observations with the clock terms of our feature maps. Both use exactly the same ARS hyperparameters and environment steps as our method. Our controllers have a higher median return than both on all six locomotion tasks, most of all on Humanoid.

Environment steps and episodes. Table 1 and Table 2 give the budget for every task. Humanoid uses 150M steps, Ant 30M, Walker2d and HalfCheetah 20M, Hopper 10M and Swimmer 5M. The other four tasks use 1,440 episodes, at most 1.45M steps.

AuthorsOpenReview ↗

Official Comment by Authors (2 of 2)

W3. Disconnected methodology.

We revised the method so that every domain follows one connected loop (Section 2, Algorithm 1). The LLM writes the program's structure, an optimizer fits its numeric parameters from return, and when a change in the environment breaks the structure, the LLM revises it. Atari now fits its constants. Bayesian optimization refits the best program's constants without any LLM call. Section 2.2 and Appendix A.1.2 describe the procedure (Algorithm 2), and Q3 below summarizes it. With 100k extra steps the refit raises Fishing Derby from 27.7 to 43.2 and Space Invaders from 445 to 562 (Table 5).

On revisions that hurt performance. The method keeps the best controller found so far and revises from it, and a revision that scores lower is discarded (Section 2.2, Algorithm 1). A worse revision therefore never replaces the best controller.

Closed-loop structural repair. Some changes in the environment break the program's structure, not only its parameters. Examples are a reversed or swapped actuator, a flipped sensor, or reversed gravity in Flappy Bird. Refitting cannot undo such a change. The LLM therefore revises the structure from the controller's episodes and return. The edit is kept only if the return does not drop, and the parameters are then refitted (Appendix A.1.3). In MuJoCo this restores the pre-fault score in 36 of 36 runs across six tasks, where refitting alone does not. Under reversed gravity in Flappy Bird, LLM edits repair the controller in 8 of 15 attempts (Section 3.3).

Actuator reversed

MethodHopperWalker2dHalfCheetahAnt
LLM edit3,6384,6505,3014,714
ARS refit5802,0712,5043,233
Code to Control (LLM edit + ARS)3,6384,7575,3424,746

Actuator reversed and two swapped

MethodHopperWalker2dHalfCheetahAnt
LLM edit3,6394,6565,3014,714
ARS refit6521,5243,415546
Code to Control (LLM edit + ARS)3,6564,7885,3174,698

Sensor flipped

MethodHopperWalker2dHalfCheetahAnt
LLM edit3,6394,6565,301200
ARS refit3,2041,8073,9024,338
Code to Control (LLM edit + ARS)3,6564,7975,3554,464

MuJoCo return after structural faults, mean of three seeds (Figure 4C). ARS refit uses 25% more environment steps than the original fit. Code to Control refits after LLM edits within the same budget.

Q3. Parsing and refitting constants, and PPO's observation.

The numeric literals of the program are found by parsing its syntax tree. Each nonzero literal is searched on a log scale within [θ₀/4, 4θ₀] around the LLM's value, and a zero literal within [−1, 1]. Bayesian optimization plays one episode per candidate and selects the candidate with the highest predicted score. In Atari, the new constants are kept only if they also win on held-out validation episodes (Appendix A.1.2, Algorithm 2). In Flappy Bird, each refit uses at most 100k environment steps (Section 3.3). The Atari games use the same procedure, with a refit budget equal to the game's step budget minus the steps used by synthesis, for example 87k steps on Fishing Derby. PPO is trained on the same OCAtari object coordinates as our controllers, not on raw pixels (Section 3.2, Appendix A.2).

References

[1] Mania, H., Guy, A., & Recht, B. (2018). Simple random search of static linear policies is competitive for reinforcement learning. Advances in Neural Information Processing Systems, 31.

[2] Bosio et al., Combining Large Language Models and Gradient-Free Optimization for Automatic Control Policy Synthesis. American Control Conference, 2026.

Reviewer_ooirOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes using LLMs to write code as a policy, aiming to replace directly invoking LLMs to select actions or plan within LLMs' written code-like world models. It first uses LLMs to write the initial policy code to execute in the environment, then uses the resulting trajectories to optimize the code or its numerical values, depending on whether the environment is discrete or continuous. It compares to LLM-based, Theory-based, and model-free baselines, showing superior performance on 6 Atari games, Flappy Bird, and 10 MuJoCo continuous-control tasks with less decision latency.

Critical strengths
  • The paper discusses an interesting direction of control problem, which uses programs instead of LLMs or deep neural networks as the policy to reduce the computation time and increase the performance.
Critical weaknesses

Novelty

  • The paper does not discuss other works using LLMs to evolve programs. [1] treats decision-making policies as self-evolving code, which seems similar to the proposed discrete method, and [2] writes code structure and optimizes its numerical values using a gradient-free optimizer, which seems similar to the proposed continuous method. Without discussions with other works, it is hard to interpret the novelty and benefit of the proposed methods.

Experiments

  • The paper only compares to LLM-based, Theory-based, and model-free policies, without comparison to other program-as-policy works.
  • The authors describe the proposed method as an online approach, while it compares to GIF-MCTS and CQL, which are offline.
  • In Section 3.3, the paper does not explain how they fit the numerical constants in the discrete setting.

Writing

  • The paper describes its proposed methods as containing Augmented Random Search as the derivative-free search without specifying the details of it.
  • The paper did not detail the evaluated environments, making the qualitative analysis hard to understand.
  • The paper did not detail the compared baselines, and did not explain why these baselines perform so poorly on these benchmarks.

[1] Kuang et al., Learning Game-Playing Agents with Generative Code Optimization. ICML 2025 Workshop on Programmatic Representations for Agent

[2] Bosio et al., Combining Large Language Models and Gradient-Free Optimization for Automatic Control Policy Synthesis. American Control Conference, 2026

Questions for authors
  • Why did the authors report the median instead of the mean and standard deviation over three seeds?
  • In Figure 2C, the authors report the number of environment steps when it finds the best programs, but it seems like other baselines are reported using their full evaluated budget. Does it mean that the score is the score from the last evaluation, but not the best among all?
  • How many task variants do the authors use for each environment? It's important to see how well the program can perform under the effects of different initial states.
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1 of 2)

We thank the reviewer for pointing us to two closely related works and for the detailed questions on the method, environments and baselines. We added both works as baselines, and the revision now documents every fitting procedure, environment and baseline. These changes have strengthened the paper.

W1 / W2. Comparison with Kuang et al. [1] and Bosio et al. [2].

GameCode to ControlKuang et al. [1]
Pong20.1 ± 1.019.9 ± 1.0
Space Invaders445 ± 87328 ± 194
Asterix1,906 ± 863100 ± 0
Breakout55.9 ± 42.546.4 ± 9.5
Fishing Derby27.7 ± 4.1−87.0 ± 5.3
Freeway29.4 ± 2.827.1 ± 5.1

Atari: frozen controllers on 30 held-out episodes, mean ± std over three runs (Figure 2A).

Related Work now discusses both works. Unlike [1], Code to Control fits the program's constants from return, without LLM calls. Unlike [2], in continuous control the LLM writes a feature program rather than the policy, and ARS fits a linear policy over its features. The LLM is called only to write or revise this structure, so the number of LLM calls does not grow with the fitting budget.

Kuang et al. on Atari. Code to Control scores higher on all six games. Both methods make 20 LLM calls per run, and ours uses about 65k input tokens against 0.7M–3.2M. At a larger, equal budget of 100k extra environment steps, refitting our constants without any LLM call still scores higher than their further LLM updates on all six games (Table 5).

Bosio et al. on MuJoCo. Code to Control scores at least 3× higher on five of the six locomotion tasks, and matches it on Swimmer with 25× fewer environment steps (table under Q1).

W3. Online and offline baselines.

We added online baselines: Kuang et al. [1], Bosio et al. [2], and three matched direct-policy baselines (linear ARS on raw observations, raw observations with clock terms, and random Fourier features). All of them learn online from interaction, like our method.

We include GIF-MCTS and WorldCoder as the state of the art among methods that synthesize an executable world model and plan through it. Code to Control is designed as an alternative to this approach, so they are its most direct comparison.

W4. Fitting constants in the discrete setting.

Section 2.2 and Appendix A.1.2 now describe it (Algorithm 2). The numeric constants of a candidate program are found by parsing its syntax tree. Bayesian optimization then searches for the optimal value of a constant, looking over an interval centered on the original value chosen by the LLM. The new constants replace the LLM's only if they score higher both in their fitting episode and on held-out validation episodes. The refit needs few environment steps. On Fishing Derby its search uses 87k steps and raises the score from 27.7 to 43.2, and each Flappy Bird refit uses at most 100k steps.

W5. ARS details. Appendix A.1.3 now describes the ARS procedure in full (Algorithm 3), Table 2 gives its hyperparameters and environment-step budget for every task.

W6. Environments. Appendix A.2 describes every game and task: the observation, with an example (Box 1), the action sets, and the rewards.

AuthorsOpenReview ↗

Official Comment by Authors (2 of 2)

W7. Baselines and why they perform poorly. The Discussion now analyses why each baseline fails.

  • World models. These methods must predict the next state of every object. After all 30 LLM calls, WorldCoder's model still mispredicts more than half of the observed transitions on five of the six Atari games. A policy only needs to capture the information the next decision depends on, such as where the ball is heading, and this is generally an easier synthesis task. An accurate model is also not enough. GIF-MCTS plans by sampling many random action sequences, simulating each one, and executing the best. Balancing a pendulum needs every action to correct the error left by the previous one, and a randomly sampled sequence almost never does this for long. So even with the true simulator, as the GIF-MCTS authors report, the planner keeps the pendulum up for only about 40 of 1,000 steps. A reactive controller computes each action from the current state, so it corrects these errors directly.
  • ReAct. ReAct uses the LLM itself as the policy, so the LLM must turn positions and velocities into a precise action at every step. The current LLMs can be unreliable at this kind of quantitative judgment. On Pong it reads the positions correctly but often does not move the paddle toward the ball.
  • Kuang et al. Kuang et al. face the same weakness in the program's constants. The LLM excels at writing effective control logic at a high level, but often fails with lower-level details, such as choosing well-calibrated constants. Kuang et al. change them only when the LLM rewrites the program, while Code to Control instead fits the constants from return, and refitting raises our mean score on all six Atari games (Table 5).
  • Bosio et al. We speculate that the principle reason their method underperforms ours is limited expressivity: like us, they also tune constants, but their learned programs typically only contain a handful of tuned literals, which are likely too low-dimensional for complex control tasks. By contrast, a linear policy on raw observations has many tunable parameters but cannot encode nonlinear structures, which provide greater expressivity. Our method combines the strengths of each of these.
  • PPO. PPO lacks the inductive biases needed to master our domains within the same order of magnitude as our symbolically-structured methods.

Q1. Median and mean. Atari now reports the mean ± std over three runs (Figure 2A). For MuJoCo, Appendix A.3.2 now reports the mean ± std (Table 7, below). Our controllers have the highest mean return on five of the six locomotion tasks.

TaskCode to ControlRaw ARSRaw + clockRFFBosio et al. [2]
Humanoid6,798 ± 2,8296,002 ± 2,0891,848 ± 1,0961,358 ± 290458 ± 58
Ant4,693 ± 1274,696 ± 2404,730 ± 873,728 ± 7951,088 ± 125
Walker2d4,650 ± 9903,469 ± 4234,168 ± 2504,175 ± 457888 ± 304
HalfCheetah5,301 ± 8194,671 ± 1,6675,249 ± 1633,369 ± 5901,670 ± 345
Hopper3,638 ± 1073,594 ± 493,419 ± 38480 ± 991,072 ± 34
Swimmer360.8 ± 0.9258.7 ± 135.2305.6 ± 46.8277.7 ± 110.0356.6 ± 1.6
HumanoidStandup130,510 ± 13,833132,574 ± 11,229140,814 ± 5,709129,666 ± 14,561116,952 ± 10,532
Inverted Pendulum1,000 ± 0676 ± 5611,000 ± 0235 ± 2681,000 ± 0
Inv. Double Pendulum9,358.8 ± 0.86,264 ± 5,362181 ± 105149 ± 473,500 ± 5,075
Reacher−8.93 ± 1.75−7.30 ± 1.11−8.44 ± 2.60−29.4 ± 19.2−5.32 ± 0.06

MuJoCo: test return, mean ± std over the three fits of Table 1 and Table 6 (Table 7).

Q2. Figure 2C. The score is the best among all episodes, not the last one. For every LLM-based method, Figure 2C reports the score and the interaction cost up to the episode that found its best controller, so the criterion is the same for all methods. PPO is shown at fixed budgets. The full search cost and the held-out scores are now reported separately (Section 3.2, Figure 2A), and new learning curves count every episode of the search (Appendix A.3).

Q3. Task variants and initial states. We test variation in initial states on held-out episodes. Each Atari controller plays 30 seeds, each starting after a random number of no-op actions. Each MuJoCo controller plays ten from random initial states, on seeds never used for fitting. Beyond initial states, Flappy Bird tests eight changes in dynamics (Figure 4B), and MuJoCo tests three structural faults (Figure 4C).

References

[1] Kuang et al., Learning Game-Playing Agents with Generative Code Optimization. ICML 2025 Workshop on Programmatic Representations for Agent Learning.

[2] Bosio et al., Combining Large Language Models and Gradient-Free Optimization for Automatic Control Policy Synthesis. American Control Conference, 2026.

Reviewer_d63QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Code to Control synthesizes executable Python policies that require neither LLM inference nor planning during deployment. For discrete tasks, the LLM revises controllers using rollout feedback. For continuous control, it proposes feature maps whose action-mapping weights are fitted using Augmented Random Search (ARS). Experiments cover six Atari games, Flappy Bird, and ten MuJoCo tasks, including execution latency and adaptation to changed dynamics.

Critical strengths
  • Separating program structure from numerical parameter fitting provides a practical way to construct compact, inspectable controllers. Direct execution removes repeated LLM calls and planning from action selection, and the supplied controller examples make the approach concrete.

  • The Flappy Bird experiments meaningfully distinguish zero-shot transfer from adaptation through parameter refitting with fixed program structure. Reporting both successful transfers and failures helps characterize when the learned structure remains useful and when numerical adaptation is insufficient.

Critical weaknesses
  • The Atari evaluation reports the best episode encountered during search, followed by a median across three runs (Section 3.2). This does not establish the expected performance of the selected controller on fresh episodes and remains susceptible to selection bias. Moreover, environment-step counts up to the retrospectively identified best episode do not represent the full search cost. These issues weaken the performance and interaction-efficiency comparisons with PPO.

  • The continuous-control experiments do not isolate the contribution of LLM-generated features. ARS directly optimizes the action-mapping weights, but there is no matched comparison using raw observations or a fixed feature basis with the same optimizer. Comparisons mainly against planning-based methods therefore leave unclear whether feature synthesis materially improves over simpler direct-policy optimization. The ARS hyperparameters and total interaction budget are also insufficiently specified.

Questions for authors

What are the mean returns and uncertainty estimates when the selected Atari controllers are frozen and evaluated on fresh seeds? Please also report the full search interaction cost and clarify whether PPO receives comparable observations and evaluation treatment.

How does LLM-generated feature synthesis compare with ARS on raw observations or fixed features under the same total interaction budget? Please specify the parameter-search settings, candidate-selection protocol, and evaluation repetitions.

1 comment

We thank the reviewer for a careful review and for concrete questions that point to clear experiments. We ran all of them, and they have strengthened the paper. The Atari evaluation now uses held-out episodes and reports the full search cost, and the MuJoCo experiments now include matched direct-policy baselines with the same optimizer and budget.

W1 / Q1. Held-out episodes, full search cost, and PPO.

GameCode to ControlKuang et al. (2025)PPO @100kPPO @20M
Pong20.1 ± 1.019.9 ± 1.0−19.7 ± 2.3−2.0 ± 18.5
Space Invaders445 ± 87328 ± 19470 ± 61108 ± 6
Asterix1,906 ± 863100 ± 0174 ± 62554 ± 51
Breakout55.9 ± 42.546.4 ± 9.50.3 ± 0.04.9 ± 3.1
Fishing Derby27.7 ± 4.1−87.0 ± 5.3−99.0 ± 0.0−97.0 ± 1.8
Freeway29.4 ± 2.827.1 ± 5.111.0 ± 10.921.7 ± 0.0
  • Each selected controller is now frozen and plays 30 held-out episodes. Figure 2A reports the mean ± std over three runs.
  • We added Kuang et al. [1], a direct policy-optimization method in which an LLM revises a policy program from its play. Under this held-out evaluation, Code to Control scores higher than every baseline on all six games, including PPO at both 100k and 20M environment steps.
  • The full search cost is now reported. A full search of 20 episodes takes from a median of 9k environment steps on Asterix to a median of 120k on Fishing Derby (Section 3.2). The learning curves in Appendix A.3 count every episode.
  • PPO receives the same OCAtari object state as our controllers, not pixels. Its final policy is frozen and scored on the same held-out episodes (Section 3.2, Appendix A.2).

W2 / Q2. Isolating the LLM features, a fixed basis, and the search settings.

TaskStepsCode to ControlRaw ARSRaw + clockRFFBosio et al. (2026)
Humanoid150M8,0136,9381,2331,344483
Ant30M4,7574,7044,7113,3981,132
Walker2d20M4,8803,2764,1673,9481,022
HalfCheetah20M5,7555,1865,1853,4861,485
Hopper10M3,5943,5683,4304581,060
Swimmer5M361311279320357

Test return, median over three runs (Table 1, Table 6).

  • We added four direct-policy baselines. Bosio et al. [2] treat the policy as code. An LLM evolves complete controller programs in an evolutionary loop, and an evolution strategy tunes their constants. The other three are matched ablations of our method: linear ARS on raw observations, raw observations with the clock terms of our feature maps (also requested by Reviewer 9QTP), and random Fourier features [3] of the same width as our feature maps. These three use the same optimizer, hyperparameters, environment steps and evaluation as our method.
  • Our controllers have a higher median return than linear ARS on all six locomotion tasks, most of all on Humanoid (8,013 against 6,938). The clock-term and random-feature ablations also score below ours on all six. With random Fourier features, the humanoid falls about 20% of the way into the episode. This suggests that the LLM writes terms that are useful for locomotion, which the linear policy fitted by ARS can then use. On Humanoid, for example, its features read 15% of the observation and include restoring, damping and left–right terms. Code to Control scores at least 3× higher than Bosio et al. on five of the six tasks, and matches it on Swimmer with 25× fewer environment steps.
  • Parameter-search settings. Appendix A.1.3 now describes the ARS procedure in full (Algorithm 3). Table 2 gives its hyperparameters and environment-step budget for every task.
  • Candidate-selection protocol. Every ten iterations the current weights play fixed evaluation episodes, and the best evaluated weights are kept.
  • Evaluation repetitions. For every method, the kept controller is frozen and plays ten test episodes on seeds never used for fitting. We report the median over three independent fits.

References

[1] Kuang et al., Learning Game-Playing Agents with Generative Code Optimization. ICML 2025 Workshop on Programmatic Representations for Agent Learning.

[2] Bosio et al., Combining Large Language Models and Gradient-Free Optimization for Automatic Control Policy Synthesis. American Control Conference, 2026.

[3] Rahimi, A., & Recht, B. (2007). Random features for large-scale kernel machines. Advances in Neural Information Processing Systems, 20.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank the reviewers for detailed and constructive reviews, which led to a substantially stronger paper. The main changes are:

  1. Stronger results. In the submission, some locomotion controllers mostly stood in place. With larger fitting budgets and the alive bonus removed from the fitting objective, all of them now walk, and the Humanoid covers 38 m per episode. Refitting constants in Atari raises scores further, for example on Fishing Derby from 27.7 to 43.2 (Table 5).
  2. Better evaluation. Atari controllers are now frozen and evaluated on 30 held-out episodes, with the mean ± std over three runs (Figure 2A). We report the full search cost and learning curves that count every episode (Appendix A.3).
  3. More baselines. We added two program-as-policy methods, Kuang et al. (2025) and Bosio et al. (2026), and three matched direct-policy baselines in MuJoCo: linear ARS on raw observations, raw observations with clock terms, and random Fourier features. Code to Control scores higher than Kuang et al. on all six Atari games, and has the highest mean return on five of the six locomotion tasks.
  4. More connected methodology. In every domain the LLM writes the program's structure, and an optimizer fits its numeric parameters from return. When a change in the environment breaks the structure, closed-loop structural repair lets the LLM revise it.
  5. Failure analysis. The Discussion now explains why each baseline fails.
  6. More detail and clearer writing. The method and appendix now describe each component of our system, including the fitting algorithms (Algorithms 2 and 3) and their hyperparameters. The appendix also details every environment and task.

Atari: frozen controllers on 30 held-out episodes, mean ± std over three runs (Figure 2A)

GameCode to ControlKuang et al.PPO @100kPPO @20M
Pong20.1 ± 1.019.9 ± 1.0−19.7 ± 2.3−2.0 ± 18.5
Space Invaders445 ± 87328 ± 19470 ± 61108 ± 6
Asterix1,906 ± 863100 ± 0174 ± 62554 ± 51
Breakout55.9 ± 42.546.4 ± 9.50.3 ± 0.04.9 ± 3.1
Fishing Derby27.7 ± 4.1−87.0 ± 5.3−99.0 ± 0.0−97.0 ± 1.8
Freeway29.4 ± 2.827.1 ± 5.111.0 ± 10.921.7 ± 0.0

MuJoCo locomotion: test return, mean ± std over three runs (Table 7, same optimizer settings and environment-step budget)

TaskStepsCode to ControlRaw ARSRaw + clockRFFBosio et al.
Humanoid150M6,798 ± 2,8296,002 ± 2,0891,848 ± 1,0961,358 ± 290458 ± 58
Ant30M4,693 ± 1274,696 ± 2404,730 ± 873,728 ± 7951,088 ± 125
Walker2d20M4,650 ± 9903,469 ± 4234,168 ± 2504,175 ± 457888 ± 304
HalfCheetah20M5,301 ± 8194,671 ± 1,6675,249 ± 1633,369 ± 5901,670 ± 345
Hopper10M3,638 ± 1073,594 ± 493,419 ± 38480 ± 991,072 ± 34
Swimmer5M360.8 ± 0.9258.7 ± 135.2305.6 ± 46.8277.7 ± 110.0356.6 ± 1.6

MuJoCo: return after structural faults, mean of three seeds (Figure 4C)

FaultMethodHopperWalker2dHalfCheetahAnt
Actuator reversedARS refit5802,0712,5043,233
Code to Control3,6384,7575,3424,746
Reversed + two swappedARS refit6521,5243,415546
Code to Control3,6564,7885,3174,698
Sensor flippedARS refit3,2041,8073,9024,338
Code to Control3,6564,7975,3554,464

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.