- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies task-agnostic environment preprocessing: an agent explores an unfamiliar environment and prepares reusable artifacts before seeing any downstream tasks. The authors introduce two open-ended META-AGENT variants, one with access to a workflow archive, and compare them with PREPING and CORPUS2SKILL while keeping the downstream solver fixed. Across six benchmarks, preprocessing improves performance in 21 of 24 method–benchmark comparisons. A META-AGENT type performs better on five benchmarks and CORPUS2SKILL performs better on the corpus-intensive BCP-G case. The evaluation also focuses on archive exploitation, migration to alternative solvers, and the trade-off between preparation cost and conducting multiple test cases. The main contribution is a general framework and empirical study of environment preparation, with evidence that its benefits depend on both the environment and the solver that consumes the artifacts.
Critical strengths
- The task-agnostic setting gives the paper a clear research question. The preprocessing agent has to prepare useful resources without seeing the downstream tasks or receiving task-derived feedback. Freezing the artifacts before evaluation makes the scope of preparation explicit. I find this setting useful to examine what can be learned by an agent from an environment in isolation and whether such preparation facilitates a fixed solver afterwards. The contribution is a well-defined problem and a practical way to study it.
- The experiments address when preparation is worth doing. The six benchmarks cover substantially different environments, and the cost and test-time scaling analyses give the results practical context. I particularly appreciate the cases where the general meta-agent is not the strongest method, as well as the transfer experiment showing that artifacts can be less useful to another solver. These results help establish the limits of the approach. The break-even analysis also makes the preparation cost easier to assess for settings where an environment will support many tasks.
Critical weaknesses
Such comparison fails to disentangle the contribution of study design from the cost of resources used. There are various methods for different base models and different budgets. Although reporting costs is helpful, we still need to identify how much of the relative improvement is due to our orchestration strategy. The method that integrates the archive also contains design, workflow definition and artifact creation in addition to selection, hence the advantage of it versus selection alone is not evident. A controlled comparison based on the same base model and a comparable study budget would further strengthen the claim that open-ended studying is more effective than fixed pipelines.
The confidence in the magnitude of method rankings is also questionable. Each method only provides three independent study runs, which cannot demonstrate the variance of its performance among study runs well. Additionally, the provided significance tests do not consider multiple comparisons. I would thus differentiate the general conclusion that preparation facilitates from the more robust conclusion that one particular study method is superior to another. The latter one needs a more detailed report on how much variability there is between different study runs and which comparisons are still significant after correcting multiplicity.
Questions for authors
How much of the advantage over fixed pipelines remains when the underlying model and total study budget are matched? A controlled comparison on a representative subset of environments would help distinguish improvements from the study strategy from improvements due to additional computation or model choice.
What exactly is resampled in the bootstrap analysis? In particular, do the intervals account for variability across independently generated study artifacts, as well as variability across downstream tasks and solver rollouts? Could you also report which method comparisons remain significant after multiple-comparison correction?
With current workflow execution records, how can we obtain the selector baseline? How are those runs assigned to the selector, and how is their variability represented in the comparison? This would help assess whether the archive-assisted agent gains from selecting a suitable workflow or from executing and combining workflows beyond that selection.
2 comments
Looking Forward To Your Response
Thanks for your review! As the response period is ending soon, we would really appreciate if the reviewer can take a look at our responses and let us know if the concerns are addressed as we have conducted further experiments at their request. Thank you!
We thank the reviewer for identifying our contribution as being well defined and practical. We address their concerns below.
W1. Separating study design from model, budget and workflow selection
Q1. Matching base model and study budget
Addressed in General Response (1/5).
Q3. Archive agent versus selection alone
The construction of the selector baseline is described in Appendix A.2.2. The selector receives a text description of the environment and of each archive workflow, and chooses one workflow without running it or seeing any artifacts. We score each choice with the main-experiment arm for that workflow, so the selector rows contain no new rollouts. The reported SD applies the law of total variance across selections and study runs. Meta-Agent w/ Archive exceeds the selector on five of six benchmarks, by up to 0.145 Avg@3 on DABStep, and the exception is BCP-G, where the selector picks Corpus2Skill. A fixed rule that runs every archive workflow and delivers all their outputs also scores below Meta-Agent w/ Archive on the three benchmarks we tested (response to Reviewer EEAw, W1 Q1), so its gain is not explained by selection alone or by running every workflow.
W2. Confidence in method rankings
Q2. Bootstrapping and multiple-comparison correction
In the submission, the intervals in Tables 6 and 7 resampled tasks only. Each task's score combined its rollouts within each study run, so rollout variability was included, but the three artifact sets were held fixed, so variation across study runs was not. Only Table 8 also resampled study runs.
We reran every comparison with a bootstrap that resamples both tasks and, for each method, its three study runs (within each world for APEX-Agents). Rollouts stay attached to their study run, so their variability still enters through the per-task scores. We apply Holm correction at α=0.05 within each family of comparisons: the 24 primary and 24 transfer comparisons against No Study, and the 6 archive comparisons in Table 8.
After correction, 20 of the 24 primary Avg@3 comparisons remain significant gains over No Study (21 in the submission). After solver transfer, 2 gains and 4 losses remain significant (7 and 5 in the submission). The archive gains on BCP-G and DABStep in Table 8 remain significant. The full procedure and revised tables are in General Response (2/5)–(4/5).