- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes EnvLunch, an annotation-free reinforcement learning warm-up procedure for language-model agents. The method samples executable SQL or pandas programs, executes them to obtain target observations, and trains an agent to recover a program that reproduces those observations through interaction with the environment. To discourage hardcoding the provided outputs, candidate programs are evaluated against the reference programs on hidden, perturbed copies of the environment. The resulting checkpoint is subsequently fine-tuned on annotated downstream tasks using GRPO.
Experiments with Qwen3.5-4B and Qwen3.5-9B report average improvements of approximately 9.6 and 9.3 percentage points across training budgets on six data-analysis benchmarks. Additional experiments examine transfer to seven other benchmarks, alternative warm-up objectives, reward hacking, and changes in information-seeking behavior.
Critical strengths
-
The paper presents a clear and practical approach to constructing agent-training tasks directly from executable environments without human annotation.
-
Experiments show consistent improvements across six data-analysis benchmarks and two model scales, with substantial gains in low-label settings.
-
Counterfactual verification effectively mitigates constant-output reward hacking, supported by a clear ablation study.
-
Alternative warm-up comparisons and behavioral analyses provide useful evidence about the method’s benefits and its effects on information-seeking behavior.
Critical weaknesses
-
Label-efficiency evaluation: The evaluation increases label budgets and training steps together. It remains unclear whether the baseline could achieve comparable performance by training longer on the same labeled subset.
-
Novelty and attribution: The method is closely related to execution-grounded program induction, and its methodological novelty requires clearer justification. The comparisons do not isolate the benefit of removing natural-language task descriptions.
-
Limited environment coverage: The warm-up procedure is evaluated only on databases and dataframes using manually designed operators and perturbations. Its applicability to other types of environments remains unverified.
-
Limited model scales: Experiments are restricted to Qwen3.5-4B and Qwen3.5-9B. Whether EnvLunch retains its advantage over self-play and LLM-generated-task RL with larger, more capable models remains unclear.
Questions for authors
- Could you compare EnvLunch and the baseline using fixed labeled subsets while varying the number of training steps? Do the label-efficiency gains persist when the baseline trains longer on the same data?
- Could you evaluate larger, more capable models to determine whether EnvLunch retains its advantage over self-play and RL on LLM-generated tasks?
2 comments
Response to Reviewer h6oi Part 2
Q3. The comparisons do not isolate the benefit of removing natural-language task descriptions.
R3: Removing natural-language task descriptions is simply an observable consequence of the EnvLunch design. The direct benefit of removing task descriptions is only that human annotation is no longer required.
The underlying design is that EnvLunch requires the model to observe the final state reached by a random trajectory and recover that trajectory. This process does not require additional task descriptions. Such a design can force the model to explore the environment while maintaining verifiability.
We have added two more detailed ablation experiments to help understand the advantages of EnvLunch:
-
Hindsight relabeling: Let the model explore autonomously instead of using random combinations of actions, while keeping all other settings exactly the same as EnvLunch. Thus, the warm-up task still requires the model to observe the final state, but it now needs to recover the trajectory generated by its own exploration.
-
Generated questions: Generate a question based on the final state reached by a random combination of actions, and verify the question, instead of directly providing the state and asking the model to recover the trajectory. All other settings remain exactly the same as EnvLunch.
These two experiments provide more detailed ablations of the EnvLunch design from two different perspectives. Both perform worse than the original EnvLunch. We hope these results address your concerns.
ConvQA
| Method | 16 | 32 | 64 | 128 | 256 | 512 | 1024 | Average |
|---|---|---|---|---|---|---|---|---|
| Baseline | 49.26 | 55.34 | 65.97 | 74.16 | 76.16 | 76.25 | 77.32 | 67.78 |
| EnvLunch | 69.67 | 71.06 | 73.64 | 76.29 | 78.87 | 79.47 | 82.21 | 75.89 |
| Hindsight relabeling | 65.19 | 67.74 | 71.11 | 73.92 | 75.91 | 79.27 | 79.16 | 73.19 |
| Generated questions | 62.92 | 64.13 | 69.45 | 77.59 | 78.73 | 74.63 | 67.36 | 70.69 |
TAT-QA
| Method | 16 | 32 | 64 | 128 | 256 | 512 | 1024 | Average |
|---|---|---|---|---|---|---|---|---|
| Baseline | 43.39 | 70.38 | 82.93 | 88.60 | 88.19 | 88.12 | 90.69 | 78.90 |
| EnvLunch | 88.12 | 89.20 | 89.61 | 89.34 | 90.08 | 90.35 | 90.69 | 89.63 |
| Hindsight relabeling | 82.66 | 82.93 | 84.48 | 87.11 | 88.26 | 87.05 | 88.60 | 85.87 |
| Generated questions | 69.23 | 77.53 | 88.06 | 90.69 | 89.27 | 88.87 | 72.27 | 82.27 |
Q4. The warm-up procedure is evaluated only on databases and dataframes using manually designed operators and perturbations.
R4: The operators are not entirely manually designed. The operators themselves are the set of interactive actions provided by the environment.
The workflow and underlying principles of EnvLunch remain exactly the same across different types of environments. We only need to replace the action pool according to the specific environment.
We have added experiments in which EnvLunch is used to warm up a 9B model directly in AlfWorld and AppWorld, respectively. These results demonstrate that EnvLunch remains effective:
AppWorld
| DownStream Training Samples | 8 | 16 | 32 | 64 |
|---|---|---|---|---|
| Baseline | 6.5 | 4.1 | 5.3 | 1.2 |
| EnvLunch, data analysis domain warm-up (paper) | 7.6 | 10.0 | 10.5 | 11.7 |
| EnvLunch, AppWorld warm-up | 8.8 | 7.0 | 10.5 | 14.0 |
ALFWorld
| DownStream Training Samples | 16 | 32 | 64 | 128 | 256 | 512 | 1024 |
|---|---|---|---|---|---|---|---|
| Baseline | 34.8 | 37.6 | 49.7 | 70.9 | 80.1 | 88.3 | 95.5 |
| EnvLunch, data analysis environment warm-up (paper) | 54.7 | 53.0 | 57.2 | 61.4 | 81.3 | 84.8 | 90.0 |
| EnvLunch, AlfWorld warm-up | 55.2 | 56.5 | 64.7 | 72.4 | 83.6 | 93.0 | 97.0 |
Q5. Experiments are restricted to Qwen3.5-4B and Qwen3.5-9B.
R5: Due to our available computational resources, we can currently only conduct reinforcement learning experiments with models at the 4B and 9B scales.
However, as we can see, when scaling up from 4B to 9B, EnvLunch remains effective and achieves better results in the cross-domain experiments. In the future, we will extend this approach to larger models and more general agentic scenarios.
Response to Reviewer h6oi Part 1
Thank you very much for your positive assessment of our work! We hope that our responses address your concerns.
Q1. The evaluation increases label budgets and training steps together.
R1: We randomly sampled 16 and 64 downstream labeled examples and conducted additional experiments following the same settings as the single-example experiments. The results are shown below. EnvLunch still demonstrates advantages over the baseline under the same number of training steps, even with a sufficient amount of data.
16 examples
64 examples
Q2. The method is closely related to execution-grounded program induction, and its methodological novelty requires clearer justification.
R2: We agree that recovering a program from its output is a classical problem, and we build on that line of work as stated in the Related Works section. It establishes that an output constrains the program that produces it and that instance-equivalent programs need to be ruled out.
Our contribution is not a new program-induction algorithm. It is the observation that this problem, posed by the environment itself, provides a source of unlimited and verifiable agentic training tasks that require no human- or model-written questions.
Concretely:
The goal of EnvLunch is not to recover the program itself, but to learn transferable interaction behaviour. The warm-up tasks share neither environments nor task forms with the downstream benchmarks.
The agent does not receive the database and output as a static specification. Instead, it must explore the environment over multiple turns.
Instance equivalence is handled at training time by rewarding agreement on perturbed copies that the agent never sees. This turns an offline equivalence check into a reward that prevents output copying. The construction is also not specific to SQL and can be generalised to other domains (see our response below).
Thank you for this advice. We will elaborate on our contributions and methodological novelty more precisely.