acceptodds
Under review as a conference paper at ICLR 2027

EnvLunch: The Environment as a Free Lunch for Learning to Interact

Abstract

For agentic data analysis, language-model agents often require a large number of annotated tasks to learn how to interact effectively with executable environments (i.e., databases). While high-quality annotated tasks require heavy manual effort, an executable environment already provides rich interaction structure and verifiable outcomes. In this sense, the environment itself offers a free lunch, an additional set of verifiable tasks that require no human annotation. We propose EnvLunch, a warm-up stage that turns this environmental signal into agent training without human-written or model-generated task descriptions, before downstream agentic reinforcement learning. EnvLunch samples valid action trajectories, executes them to obtain final observations, and trains an agent to recover a functionally equivalent trajectory given only its final observation. This label-free warm-up enables the model to outperform the baseline by over 9% on average across all downstream training budgets, and with few labels, the baseline needs 8 to 32 times as many annotated examples to match the accuracy that the EnvLunch model reaches at its smallest training budget. Letting the environment alone define the task yields a larger gain than having the model write its own tasks over the same environments, and this warm-up also improves fine-tuning on benchmarks beyond data analysis. EnvLunch therefore saves annotated tasks and provides a better starting point for fine-tuning agentic models when such tasks are limited.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model agent, agent environment, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_h6oiOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes EnvLunch, an annotation-free reinforcement learning warm-up procedure for language-model agents. The method samples executable SQL or pandas programs, executes them to obtain target observations, and trains an agent to recover a program that reproduces those observations through interaction with the environment. To discourage hardcoding the provided outputs, candidate programs are evaluated against the reference programs on hidden, perturbed copies of the environment. The resulting checkpoint is subsequently fine-tuned on annotated downstream tasks using GRPO.

Experiments with Qwen3.5-4B and Qwen3.5-9B report average improvements of approximately 9.6 and 9.3 percentage points across training budgets on six data-analysis benchmarks. Additional experiments examine transfer to seven other benchmarks, alternative warm-up objectives, reward hacking, and changes in information-seeking behavior.

Critical strengths
  1. The paper presents a clear and practical approach to constructing agent-training tasks directly from executable environments without human annotation.

  2. Experiments show consistent improvements across six data-analysis benchmarks and two model scales, with substantial gains in low-label settings.

  3. Counterfactual verification effectively mitigates constant-output reward hacking, supported by a clear ablation study.

  4. Alternative warm-up comparisons and behavioral analyses provide useful evidence about the method’s benefits and its effects on information-seeking behavior.

Critical weaknesses
  1. Label-efficiency evaluation: The evaluation increases label budgets and training steps together. It remains unclear whether the baseline could achieve comparable performance by training longer on the same labeled subset.

  2. Novelty and attribution: The method is closely related to execution-grounded program induction, and its methodological novelty requires clearer justification. The comparisons do not isolate the benefit of removing natural-language task descriptions.

  3. Limited environment coverage: The warm-up procedure is evaluated only on databases and dataframes using manually designed operators and perturbations. Its applicability to other types of environments remains unverified.

  4. Limited model scales: Experiments are restricted to Qwen3.5-4B and Qwen3.5-9B. Whether EnvLunch retains its advantage over self-play and LLM-generated-task RL with larger, more capable models remains unclear.

Questions for authors
  1. Could you compare EnvLunch and the baseline using fixed labeled subsets while varying the number of training steps? Do the label-efficiency gains persist when the baseline trains longer on the same data?
  2. Could you evaluate larger, more capable models to determine whether EnvLunch retains its advantage over self-play and RL on LLM-generated tasks?
2 comments
AuthorsOpenReview ↗

Response to Reviewer h6oi Part 1

Thank you very much for your positive assessment of our work! We hope that our responses address your concerns.


Q1. The evaluation increases label budgets and training steps together.

R1: We randomly sampled 16 and 64 downstream labeled examples and conducted additional experiments following the same settings as the single-example experiments. The results are shown below. EnvLunch still demonstrates advantages over the baseline under the same number of training steps, even with a sufficient amount of data.

16 examples

Dataset / MethodStep 248163264128Average
ConvFinQA EnvLunch69.672.473.372.575.374.576.073.4
ConvFinQA Baseline46.154.068.171.370.370.773.764.9
TAT-QA EnvLunch87.889.188.988.589.389.588.388.8
TAT-QA Baseline40.844.573.477.177.679.277.267.1
DataBench EnvLunch74.174.876.578.280.977.278.177.1
DataBench Baseline67.167.265.769.772.467.669.168.4

64 examples

Dataset / MethodStep 248163264128Average
ConvFinQA EnvLunch68.069.370.773.075.474.974.272.2
ConvFinQA Baseline46.253.666.474.346.767.279.862.0
TAT-QA EnvLunch88.587.587.790.388.190.589.588.9
TAT-QA Baseline39.742.787.089.389.990.788.375.4
DataBench EnvLunch73.573.274.170.980.382.679.876.3
DataBench Baseline65.868.152.770.166.175.875.567.7

Q2. The method is closely related to execution-grounded program induction, and its methodological novelty requires clearer justification.

R2: We agree that recovering a program from its output is a classical problem, and we build on that line of work as stated in the Related Works section. It establishes that an output constrains the program that produces it and that instance-equivalent programs need to be ruled out.

Our contribution is not a new program-induction algorithm. It is the observation that this problem, posed by the environment itself, provides a source of unlimited and verifiable agentic training tasks that require no human- or model-written questions.

Concretely:

  1. The goal of EnvLunch is not to recover the program itself, but to learn transferable interaction behaviour. The warm-up tasks share neither environments nor task forms with the downstream benchmarks.

  2. The agent does not receive the database and output as a static specification. Instead, it must explore the environment over multiple turns.

  3. Instance equivalence is handled at training time by rewarding agreement on perturbed copies that the agent never sees. This turns an offline equivalence check into a reward that prevents output copying. The construction is also not specific to SQL and can be generalised to other domains (see our response below).

Thank you for this advice. We will elaborate on our contributions and methodological novelty more precisely.

AuthorsOpenReview ↗

Response to Reviewer h6oi Part 2

Q3. The comparisons do not isolate the benefit of removing natural-language task descriptions.

R3: Removing natural-language task descriptions is simply an observable consequence of the EnvLunch design. The direct benefit of removing task descriptions is only that human annotation is no longer required.

The underlying design is that EnvLunch requires the model to observe the final state reached by a random trajectory and recover that trajectory. This process does not require additional task descriptions. Such a design can force the model to explore the environment while maintaining verifiability.

We have added two more detailed ablation experiments to help understand the advantages of EnvLunch:

  1. Hindsight relabeling: Let the model explore autonomously instead of using random combinations of actions, while keeping all other settings exactly the same as EnvLunch. Thus, the warm-up task still requires the model to observe the final state, but it now needs to recover the trajectory generated by its own exploration.

  2. Generated questions: Generate a question based on the final state reached by a random combination of actions, and verify the question, instead of directly providing the state and asking the model to recover the trajectory. All other settings remain exactly the same as EnvLunch.

These two experiments provide more detailed ablations of the EnvLunch design from two different perspectives. Both perform worse than the original EnvLunch. We hope these results address your concerns.

ConvQA

Method1632641282565121024Average
Baseline49.2655.3465.9774.1676.1676.2577.3267.78
EnvLunch69.6771.0673.6476.2978.8779.4782.2175.89
Hindsight relabeling65.1967.7471.1173.9275.9179.2779.1673.19
Generated questions62.9264.1369.4577.5978.7374.6367.3670.69

TAT-QA

Method1632641282565121024Average
Baseline43.3970.3882.9388.6088.1988.1290.6978.90
EnvLunch88.1289.2089.6189.3490.0890.3590.6989.63
Hindsight relabeling82.6682.9384.4887.1188.2687.0588.6085.87
Generated questions69.2377.5388.0690.6989.2788.8772.2782.27

Q4. The warm-up procedure is evaluated only on databases and dataframes using manually designed operators and perturbations.

R4: The operators are not entirely manually designed. The operators themselves are the set of interactive actions provided by the environment.

The workflow and underlying principles of EnvLunch remain exactly the same across different types of environments. We only need to replace the action pool according to the specific environment.

We have added experiments in which EnvLunch is used to warm up a 9B model directly in AlfWorld and AppWorld, respectively. These results demonstrate that EnvLunch remains effective:

AppWorld

DownStream Training Samples8163264
Baseline6.54.15.31.2
EnvLunch, data analysis domain warm-up (paper)7.610.010.511.7
EnvLunch, AppWorld warm-up8.87.010.514.0

ALFWorld

DownStream Training Samples1632641282565121024
Baseline34.837.649.770.980.188.395.5
EnvLunch, data analysis environment warm-up (paper)54.753.057.261.481.384.890.0
EnvLunch, AlfWorld warm-up55.256.564.772.483.693.097.0

Q5. Experiments are restricted to Qwen3.5-4B and Qwen3.5-9B.

R5: Due to our available computational resources, we can currently only conduct reinforcement learning experiments with models at the 4B and 9B scales.

However, as we can see, when scaling up from 4B to 9B, EnvLunch remains effective and achieves better results in the cross-domain experiments. In the future, we will extend this approach to larger models and more general agentic scenarios.

Reviewer_AK3xOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EnvLunch introduces an annotation-free RL warm-up for language-model agents. It samples executable trajectories in databases and dataframes, then trains agents to recover equivalent programs from their outputs. Counterfactual environments discourage hardcoded answers. Experiments on two Qwen3.5 models show improved downstream performance across training budgets, with additional transfer evaluations and comparisons against alternative warm-up methods.

Critical strengths
  1. Output-conditioned program recovery provides automatically verifiable agent training without human-written questions or teacher-generated demonstrations, offering a practical source of additional supervision.
  2. Six primary benchmarks, seven additional tasks, and three downstream training seeds provide meaningful evidence of transfer.
  3. Counterfactual verification prevents straightforward answer copying. The corresponding ablation demonstrates that naive output matching can actively degrade performance.
Critical weaknesses
  1. The central label-efficiency claim is not sufficiently established. Labeled-data budgets and optimization steps increase together in the main experiments. The single-example analysis does not replace comparisons using repeated training on fixed small datasets, leaving the reported 8–32× savings insufficiently validated.
  2. The source of improvement remains insufficiently isolated. Synthetic-question baselines introduce potentially noisy question–answer supervision. Their weaker results therefore do not clearly distinguish the benefit of output-conditioned recovery from that of more reliable training targets. A comparison using the same underlying programs and verification would be more decisive.
  3. Functional-equivalence verification has limited validation. The counterfactual ablation convincingly addresses constant-output cheating, but does not measure acceptance of subtler incorrect programs. With only two or three perturbed environments, the reliability of this central reward mechanism remains uncertain.
Questions for authors

NA

1 comment
AuthorsOpenReview ↗

Response to Reviewer AK3x Part 1

We appreciate your recognition of our work! We hope that our responses address your concerns.

Q1. The central label-efficiency claim is not sufficiently established.

R1: We randomly sampled 16 and 64 downstream labeled examples and conducted additional experiments following the same settings as the single-example experiments. The results are shown below. Even with a sufficient amount of data, EnvLunch still demonstrates advantages over the baseline under the same number of training steps.

16 examples

Dataset / MethodStep 248163264128Average
ConvFinQA EnvLunch69.672.473.372.575.374.576.073.4
ConvFinQA Baseline46.154.068.171.370.370.773.764.9
TAT-QA EnvLunch87.889.188.988.589.389.588.388.8
TAT-QA Baseline40.844.573.477.177.679.277.267.1
DataBench EnvLunch74.174.876.578.280.977.278.177.1
DataBench Baseline67.167.265.769.772.467.669.168.4

64 examples

Dataset / MethodStep 248163264128Average
ConvFinQA EnvLunch68.069.370.773.075.474.974.272.2
ConvFinQA Baseline46.253.666.474.346.767.279.862.0
TAT-QA EnvLunch88.587.587.790.388.190.589.588.9
TAT-QA Baseline39.742.787.089.389.990.788.375.4
DataBench EnvLunch73.573.274.170.980.382.679.876.3
DataBench Baseline65.868.152.770.166.175.875.567.7

Q2. The source of improvement remains insufficiently isolated.

R2: The design of EnvLunch is based on output-conditioned recovery, which naturally brings verifiable results and reliable training targets. We did not deliberately separate these into two modules. They are both natural benefits of the EnvLunch idea of learning from the environment itself.

However, we agree that more detailed ablations can help us better understand why EnvLunch works. We have added two experiments:

  1. Hindsight relabeling: Let the model explore autonomously instead of using random combinations of actions, while keeping all other settings exactly the same as EnvLunch. Thus, the warm-up task still requires the model to observe the final state, but it now needs to recover the trajectory generated by its own exploration.

  2. Generated questions: Generate a question based on the final state reached by a random combination of actions, and verify the question, instead of directly providing the state and asking the model to recover the trajectory. All other settings remain exactly the same as EnvLunch.

These two experiments provide more detailed ablations of the EnvLunch design from two different perspectives. Both perform worse than the original EnvLunch. We hope these results address your concerns.

ConvQA

Method1632641282565121024Average
Baseline49.2655.3465.9774.1676.1676.2577.3267.78
EnvLunch69.6771.0673.6476.2978.8779.4782.2175.89
Hindsight relabeling65.1967.7471.1173.9275.9179.2779.1673.19
Generated questions62.9264.1369.4577.5978.7374.6367.3670.69

TAT-QA

Method1632641282565121024Average
Baseline43.3970.3882.9388.6088.1988.1290.6978.90
EnvLunch88.1289.2089.6189.3490.0890.3590.6989.63
Hindsight relabeling82.6682.9384.4887.1188.2687.0588.6085.87
Generated questions69.2377.5388.0690.6989.2788.8772.2782.27

Q3. Functional-equivalence verification has limited validation.

R3: We calculated how the hack rate changes with the number of copies kk. The hack rate refers to the proportion of cases where a model achieves full scores on all kk copies but is still actually incorrect. Specifically, we evaluate the model's answers on 20 copies. An answer is considered free of hacking only if it passes all 20 copies.

K0 (no copy)123456810
4B41.4%11.3%6.2%4.1%3.1%2.5%2.1%1.7%1.4%
9B45.2%11.2%5.9%3.6%2.3%1.7%1.3%0.9%0.6%
Overall43.4%11.3%6.0%3.9%2.7%2.1%1.7%1.3%1.0%

Without multi-copy perturbations and using only the original database, there is a very serious hacking problem (over 40%). However, using only three copies can quickly reduce the hack rate to 2%–3%, effectively suppressing hacking without requiring too much additional computation.

Reviewer_X7PuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents EnvLunch, a warm-up, RLVR-like method that trains agents to interact with databases and dataframes before fine-tuning on annotated tasks. Its premise is that executable environments can provide verifiable training targets without human- or model-written questions.

EnvLunch randomly combines valid SQL or pandas operations and executes them to produce a target result. The agent sees the environment and the result, but not the operations that produced it. It must find a functionally equivalent sequence. To prevent it from simply copying the result, the proposed sequence is also evaluated on perturbed copies of the environment.

The warm-up uses GRPO reinforcement learning, followed by downstream fine-tuning. On six data-analysis benchmarks, the paper reports average gains across training budgets of 9.6 percentage points for Qwen3.5-4B and 9.3 points for Qwen3.5-9B.

Critical strengths
  • The paper is well motivated. Using execution results as question-free RL tasks is a clever way to extract more training signal from existing databases.

  • The results are strong and consistent across the six main data-analysis benchmarks at both 4B and 9B. The gains are largest when annotated examples are scarce, and each fine-tuning setting is repeated with three seeds.

  • The perturbed-environment verifier addresses a clear failure mode: without it, the agent submits hardcoded answers, and held-out warm-up accuracy falls from 46.1 to 16.8. The other warm-up comparisons also strengthen the evaluation.

Critical weaknesses
  • The claim that the environment alone supplies the tasks needs qualification. EnvLunch does not require written questions, but it does rely on a designed action pool, trajectory filters, and database perturbations. It is unclear how much design would be needed to use this approach in less structured environments.

  • The comparison with model-generated tasks does not fully isolate why EnvLunch performs better. Those baselines can produce questions that do not match their answers, while EnvLunch starts from an executable trajectory with a verified result. The gain could therefore come from cleaner supervision rather than the absence of written questions. A comparison with equally reliable verification would help.

  • The transfer results in Section 5.2 are less consistent than the main data-analysis results. At 4B, the gain is only 0.4 percentage points on AppWorld and −0.2 on SpreadsheetBench. Since all warm-up environments are databases or dataframes, I would like more discussion of why transfer is strong for some additional tasks but limited for others.

Questions for authors
  • Can you compare against model-generated tasks whose questions are independently checked against their answer programs, with the same training budget and execution-based reward? This would help separate the benefit of omitting written questions from the benefit of cleaner supervision.

  • How sensitive are the results to the number and type of perturbed environments? Please vary the number of copies and test recovered queries on additional, independently generated perturbations to estimate how often incorrect queries still pass.

  • How would valid-action sampling and counterfactual verification work in a non-tabular environment? The additional benchmarks show transfer of the trained model, but do not test whether EnvLunch can generate warm-up tasks there.

3 comments
AuthorsOpenReview ↗

Response to Reviewer X7Pu Part 1

Thank you very much for your positive assessment of our work! We hope that our responses address your concerns.


Q1: The claim that the environment alone supplies the tasks needs qualification.

R1: The environment does provide the tasks. More specifically, no human-provided information beyond the environment itself is introduced. The action pool is also part of the environment, as it defines the interactions that the environment can accept. The trajectory filters simply filter out combinations of actions that cannot interact with the environment. Database perturbations are also operations performed on the environment itself.

All information comes from the environment itself. However, as you correctly pointed out, additional operations are required to transform this information into concrete training tasks. We will adopt a more accurate and rigorous description in the paper.


Q2. It is unclear how much design would be needed to use this approach in less structured environments. How would valid-action sampling and counterfactual verification work in a non-tabular environment?

R2: First, the workflow and underlying principles of EnvLunch remain exactly the same across environments with different levels of structure. We only need to replace the action pool according to the specific environment. For non-tabular environment benchmarks such as AppWorld and ALFWorld, the process is even simpler. Multiple perturbed copies are unnecessary to guard against reward hacking, since actions in tabular environments have a tricky property: they can directly set or even create ground-truth states to manipulate the reward. This is not possible in AppWorld or AlfWorld. Therefore, we only need to execute randomly composed action sequences and ask the model to recover the original action sequence from the resulting final state.

We have added experiments in which EnvLunch is used to warm up a 9B model directly in AlfWorld and AppWorld, respectively. The results are as follows.

AppWorld

DownStream Training Samples8163264
Baseline6.54.15.31.2
EnvLunch, data analysis domain warm-up (paper)7.610.010.511.7
EnvLunch, AppWorld warm-up8.87.010.514.0

AlfWorld

DownStream Training Samples1632641282565121024
Baseline34.837.649.770.980.188.395.5
EnvLunch, data analysis environment warm-up (paper)54.753.057.261.481.384.890.0
EnvLunch, AlfWorld warm-up55.256.564.772.483.693.097.0

On the other hand, in less structured environments, both the exploration space and the action pool become significantly larger. We believe EnvLunch remains useful in these settings, but it requires scaling up. Specifically, this means using longer and more complex random action combinations to cover a larger exploration space, as well as a stronger base model to support this warm-up process.

Due to hardware limitations, we have only conducted experiments with relatively small models so far. In the future, we will extend this approach to larger models and more general agentic scenarios.

AuthorsOpenReview ↗

Response to Reviewer X7Pu Part 2

Q3: The comparison with model-generated tasks does not fully isolate why EnvLunch performs better.

R3: This is a very insightful observation! In fact, the two characteristics of not requiring question generation and having verifiable results are simply natural consequences of the EnvLunch approach. We did not deliberately design them as two separate modules.

Since the result is the final environment state reached by a random combination of actions, it is naturally verifiable. Since the question is simply how to reach this state, there is no need to generate an additional question, and the question is naturally consistent with the answer.

However, as you pointed out, conducting more detailed ablations would help us better understand the underlying mechanisms of EnvLunch. Generated questions might not encourage the model to explore the environment sufficiently. Furthermore, whether the generated questions are consistent with their results has not been verified, which could introduce noise into warm-up training.

Therefore, we have added two experiments:

  1. Hindsight relabeling: Let the model explore autonomously instead of using random combinations of actions, while keeping all other settings exactly the same as EnvLunch. Thus, the warm-up task still requires the model to observe the final state, but it now needs to recover the trajectory generated by its own exploration.

  2. Generated questions: Generate a question based on the final state reached by a random combination of actions, and verify the question, instead of directly providing the state and asking the model to recover the trajectory. All other settings remain exactly the same as EnvLunch.

These two experiments provide more detailed ablations of the EnvLunch design from two different perspectives. Both perform worse than the original EnvLunch. We hope these results address your concerns.

ConvQA

Method1632641282565121024Average
Baseline49.2655.3465.9774.1676.1676.2577.3267.78
EnvLunch69.6771.0673.6476.2978.8779.4782.2175.89
Hindsight relabeling65.1967.7471.1173.9275.9179.2779.1673.19
Generated questions62.9264.1369.4577.5978.7374.6367.3670.69

TAT-QA

Method1632641282565121024Average
Baseline43.3970.3882.9388.6088.1988.1290.6978.90
EnvLunch88.1289.2089.6189.3490.0890.3590.6989.63
Hindsight relabeling82.6682.9384.4887.1188.2687.0588.6085.87
Generated questions69.2377.5388.0690.6989.2788.8772.2782.27

Q4. The transfer results in Section 5.2 are less consistent than the main data-analysis results.

R4: We will add more explanations in the paper. The main issue is that AppWorld and SpreadsheetBench, although SpreadsheetBench sounds like a data analysis task, are substantially different from all the data analysis domains used for warm-up.

This is reflected in two aspects:

  1. Different action sets: Data analysis involves various SQL and Pandas operations, whereas AppWorld involves calling smartphone APIs, and SpreadsheetBench uses openpyxl to read and write Excel files.

  2. Different difficulty levels: The untrained Qwen model can achieve an initial accuracy of over 40% on most data analysis benchmarks, whereas its accuracy on AppWorld and SpreadsheetBench is generally below 10%.

When selecting cross-domain benchmarks for analysis, we deliberately chose domains with different degrees of difference from the warm-up domain, in order to thoroughly analyse the generalisation ability of EnvLunch models. AppWorld and SpreadsheetBench are precisely the benchmarks with larger cross-domain differences.

For smaller models, such a large domain gap may be too challenging. Even so, although the 4B model does not perform well, the 9B model still achieves improvements on both benchmarks. This is particularly meaningful given that the action sets are completely different. These results effectively demonstrate the generalisation ability of EnvLunch.

AuthorsOpenReview ↗

Response to Reviewer X7Pu Part 3

Q5. How sensitive are the results to the number and type of perturbed environments?

R5: This is a very good suggestion! We analysed the warm-up data and calculated how the hack rate changes with the number of copies kk. The hack rate refers to the proportion of cases where a model achieves full scores on all kk copies but is still actually incorrect. Specifically, we evaluate the model's answers on 20 copies. An answer is considered free of hacking only if it passes all 20 copies.

The results are as follows:

K0 (no copy)123456810
4B41.4%11.3%6.2%4.1%3.1%2.5%2.1%1.7%1.4%
9B45.2%11.2%5.9%3.6%2.3%1.7%1.3%0.9%0.6%
Overall43.4%11.3%6.0%3.9%2.7%2.1%1.7%1.3%1.0%

As we can see, without multi-copy perturbations and using only the original database, there is a very serious hacking problem (over 40%). However, using only three copies can quickly reduce the hack rate to 2%–3%, effectively suppressing hacking without requiring too much additional computation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.