acceptodds
Under review as a conference paper at ICLR 2027

FARL: A Fully Automated Pipeline of RL Interface and Solver Design

Abstract

Existing reinforcement learning (RL) largely focuses on optimizing learning algorithms within a pre-defined task formulation, while applying RL to new tasks still relies heavily on human expertise to design the learning pipeline: what an agent observes, how it is rewarded, which algorithm it uses, and how training is configured. Automating these choices is challenging because they are tightly coupled. We introduce FARL (Fully Automated Reinforcement Learning), a framework that automates RL design by jointly searching observation and reward programs, learning algorithms, and training settings. FARL uses training statistics and observed behavior to diagnose failures, propose changes, and test whether they improve learning. Its search preserves diverse candidates and the controls needed for these tests, using a staged budget to screen designs before investing in longer training. Experimental memory records each finding together with its conditions, so an unsuccessful component can be reconsidered in a different combination. Across expert-withheld and expert-available settings, FARL outperforms the evaluated automated baselines on challenging navigation, locomotion, and manipulation tasks. On Adroit Hand Relocate, FARL achieves 97.1% success at 5M training steps per policy, compared with 0% for Eureka and LiMEN and 1.0% for native Proximal Policy Optimization (PPO) under the same policy training budget.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Models, Automated Machine Learning, Reward Design, Hyperparameter Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_eF1MOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents FARL, an automated RL design framework that jointly searches observation and reward programs, learning algorithms, hyperparameters, and exploration strategies. FARL uses training statistics and behavior to diagnose failures, preserves diverse candidate designs, and maintains an experimental memory to guide later decisions.

Critical strengths

The paper tackles an ambitious and important AutoRL problem. Rather than optimizing only rewards or hyperparameters, FARL jointly considers the RL interface and solver, which better reflects how RL systems are designed in practice.

Critical weaknesses

Experiments are pretty simple and cannot serve as a strong evidence to support the proposed automated RL pipeline. Instead of systematically verify the pipeline on the whole benchmark, the paper instead select one single task from each benchmark, such as selecting DoorKey as navigation task, Ant from MuJuCO as locomotion task, and DoorOpen from Meta World as manipulation task. It is very suspicious to be a cherry-picking behavior. Additionally, the total number of tasks is inadequate and the difficulty of these task are relatively simple for RL control as they are all state-based.

Questions for authors

In experiment, I notice we did not use the same interaction budget across baselines and FARL. It is not clear how much of FARL's advantage comes from the framework itself versus greater interaction budget. Could you clarify or provide more experiment result?

Reviewer_W1yLOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper is on fully automated reinforcement learning (FARL), which aims to automate RL design.

Critical strengths

I found this paper very hard to parse. See Weaknesses.

Critical weaknesses

The paper is dense and essentially unreadable, so I cannot tell what it is about / follow the argumentation. As such, I cannot provide a detailed review and recommend rejection because of poor intelligibility.

Questions for authors

N/A

Reviewer_BTdtOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents FARL, an automated reinforcement learning (AutoRL) framework that, concretely, uses a large language model to propose and revise training configurations for a supplied task. It searches over observation features, reward programs, learning algorithms and their settings, exploration procedures, and training duration. Numerical hyperparameters are searched using Optuna within domains appropriate to the selected learner. Candidate configurations are used to train policies, which are assessed using a fixed task evaluator. The LLM examines training statistics, recorded trajectories, and previous experiments to propose changes; the search controller decides which candidates to retain and train further. The authors motivate this joint search by the interaction between training choices: a reward or observation feature that fails with one learner may work with another. FARL can test proposed explanations through controlled comparisons and record the conditions under which a change helped or hurt; other search steps may change several components together. I understand the underlying problem as automated configuration search, including program-valued choices as well as numerical hyperparameters; I view the main methodological contribution as how the system directs that search. The task interface, goal, and evaluator are supplied before search.

FARL keeps observation features, reward programs, the learning algorithm, and training settings open to revision throughout search. It reserves candidates needed for controlled comparisons and preserves promising alternatives instead of selecting solely by their latest score. Existing automated reward design, interface search, and learner selection are foundations; the addition is a system that can revisit these choices using recorded experiments. For diagnostic comparisons, the system specifies a prediction, the changed and fixed components, and the training conditions, then records the outcome. Changing the learner can require different valid hyperparameter settings; not every comparison holds every other choice identical. This can prevent treating a failure under particular settings as a universal verdict on a component. The contribution is an implemented workflow and its evidence, not a new reinforcement-learning update or a guarantee of reliable automated scientific reasoning. Optional policy relay lets an existing policy reach an intermediate state before a new learner takes over; final evaluation uses the new policy alone from normal task starts.

Evaluation uses MiniGrid DoorKey, a small grid-world key-and-door task; MuJoCo Ant locomotion; Meta-World door opening (Door); and Adroit hand-object relocation (Hand). Some settings withhold task-provided reward and engineered observation designs; others permit them, while goals and public interfaces remain supplied. Comparisons include Eureka and LiMEN, automated design baselines that keep the learning algorithm fixed. Additional training runs of selected DoorKey and Ant designs support recipe repeatability, not repeatability of the whole search. The reported selected DoorKey policies succeed in all their evaluation episodes and Ant travels farther than the tested automated baselines. Selected Hand policies successfully move an object to its target and hold it there, but their evaluations also guided selection. Ant and Hand use more total search training than the baselines. In Hand, FARL predicts that removing a particular set of added hand/object/target relational features, while retaining other generated distance features and the original inputs, will improve moving the object after lifting it. With reward, learner, training settings, budget, and seeds held fixed, transport instead worsens. FARL records the failed prediction, retains both representations, and later finds settings under which both improve. This demonstrates experimental operation, not that memory caused later success. Simplified searches either fix the learner and its hyperparameters and exploration procedure (A1), remove detailed diagnostic evidence and memory (A2), or remove diversity and deliberate revival of discarded designs (A3). A2 and A3 retain access to all design choices. Full FARL solves Door where these restricted searches fail, but uncertainty about the resource limits and stopping decisions governing the comparative ablation searches restricts attribution of the broader gains. The experiments establish effective discovered designs and a functioning experimental loop more firmly than a repeatable advantage of that loop over simpler joint search.

Critical strengths

FARL can discovers effective learning configurations for chosen benchmarks. Hand outperforms the tested automated baselines when each final policy receives the same amount of training and all methods receive the same public observation fields. Its evaluations also guide selection, and FARL spends more on the preceding search. DoorKey succeeds in additional training runs at roughly Eureka's total interaction expenditure; separately trained Ant policies also support the selected design. These results demonstrate useful designs even though they do not estimate how reliably a fresh full search would discover them.

Critical weaknesses
  1. The conceptual contribution is insufficiently distinguished from automated configuration search. Each component of the proposed learning-design tuple is a choice about how to train a policy: observation features, reward code, learning algorithm, learner settings, exploration, or training duration. These can be understood as a hyperparameter search space that includes program-valued choices and settings conditional on the chosen learner. The authors acknowledge automated reinforcement learning and motivate revisiting coupled choices when training fails. That motivates joint search, but does not establish a new optimization problem. The tuple is a reasonable description of the chosen search space; I do not find a clear conceptual advance in that description itself. The methodological contribution therefore rests on how diagnosis, recorded experiments, preservation of alternatives, and training allocation improve search. The paper does not distinguish that advance from the breadth of its configurable pipeline convincingly enough.

  2. The experiments show successful configurations more clearly than they explain why the proposed search procedure finds them. Policy learning curves mainly establish how well a selected configuration trains. They do not establish whether the controller's diagnosis or memory helped discover it, or whether a simpler search with the same available choices would do as well. The Hand example is informative: a prediction about removing distance features is contradicted by a controlled test, recorded, and followed by improvements to both representations. This establishes that the experimental workflow operates, but does not establish that the diagnosis or its recorded outcome enabled those improvements. These examples show that the workflow operates, but teach me little about when or why its distinctive search rules improve discovery.

Questions for authors
  1. I understand FARL as automated configuration search over observation programs, rewards, learners, and training settings. Beyond expanding the available choices, which search decisions distinguish FARL from existing approaches? Please explain what your diagnosis, memory, or preservation rules enable that those approaches do not.

  2. Given the evaluated benchmarks are well-established: to what extent do the results reflect the search procedure, rather than the pretrained LLM supplying effective configurations for familiar benchmarks? Does the search history show meaningful improvement over the LLM’s initial proposals, and what evidence distinguishes information learned from experiments from prior knowledge of these tasks?

  3. How does FARL compare with simpler search methods given access to the same configurable components and comparable resources? A comparison with random search, Bayesian hyperparameter tuning, or population-based training adapted to this search space, would help establish whether the proposed diagnosis, memory, and preservation rules improve discovery.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.