acceptodds
Under review as a conference paper at ICLR 2027

Studying Without a Syllabus: Task-Agnostic Environment Preprocessing

Abstract

Before an LLM agent tackles tasks in a new environment, the environment's corpora and tools can be inspected to construct reusable resources such as indices, scripts, or procedural guidance. Most automated adaptation methods rely on task examples, trajectories, or evaluation feedback to decide what to build. Existing task-agnostic approaches avoid this supervision but commit in advance to strategies designed for particular environment types. We ask whether, before test time and without knowledge of the downstream task distribution, a meta-agent can instead study an unfamiliar environment without a syllabus and choose how to prepare it. We formalize task-agnostic environment preprocessing and compare unaided and archive-equipped meta-agents with fixed synthetic-practice and corpus-processing methods across six heterogeneous benchmarks. Studying significantly improves downstream performance in 21 of 24 method–benchmark comparisons. A meta-agent variant leads on five benchmarks, while fixed corpus processing leads on the most corpus-heavy benchmark. The same artifacts provide less consistent gains when used by a different task-solving agent. Studied artifacts can also reduce the sampling needed to reach a given score, showing how reusable preparation can shift computation from repeated test-time attempts to pre-task study.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Learning to Learn, Exploration, Information Retrieval, Planning, Memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_tkqAOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies task-agnostic environment preprocessing: an agent explores an unfamiliar environment and prepares reusable artifacts before seeing any downstream tasks. The authors introduce two open-ended META-AGENT variants, one with access to a workflow archive, and compare them with PREPING and CORPUS2SKILL while keeping the downstream solver fixed. Across six benchmarks, preprocessing improves performance in 21 of 24 method–benchmark comparisons. A META-AGENT type performs better on five benchmarks and CORPUS2SKILL performs better on the corpus-intensive BCP-G case. The evaluation also focuses on archive exploitation, migration to alternative solvers, and the trade-off between preparation cost and conducting multiple test cases. The main contribution is a general framework and empirical study of environment preparation, with evidence that its benefits depend on both the environment and the solver that consumes the artifacts.

Critical strengths
  1. The task-agnostic setting gives the paper a clear research question. The preprocessing agent has to prepare useful resources without seeing the downstream tasks or receiving task-derived feedback. Freezing the artifacts before evaluation makes the scope of preparation explicit. I find this setting useful to examine what can be learned by an agent from an environment in isolation and whether such preparation facilitates a fixed solver afterwards. The contribution is a well-defined problem and a practical way to study it.
  2. The experiments address when preparation is worth doing. The six benchmarks cover substantially different environments, and the cost and test-time scaling analyses give the results practical context. I particularly appreciate the cases where the general meta-agent is not the strongest method, as well as the transfer experiment showing that artifacts can be less useful to another solver. These results help establish the limits of the approach. The break-even analysis also makes the preparation cost easier to assess for settings where an environment will support many tasks.
Critical weaknesses

Such comparison fails to disentangle the contribution of study design from the cost of resources used. There are various methods for different base models and different budgets. Although reporting costs is helpful, we still need to identify how much of the relative improvement is due to our orchestration strategy. The method that integrates the archive also contains design, workflow definition and artifact creation in addition to selection, hence the advantage of it versus selection alone is not evident. A controlled comparison based on the same base model and a comparable study budget would further strengthen the claim that open-ended studying is more effective than fixed pipelines.

The confidence in the magnitude of method rankings is also questionable. Each method only provides three independent study runs, which cannot demonstrate the variance of its performance among study runs well. Additionally, the provided significance tests do not consider multiple comparisons. I would thus differentiate the general conclusion that preparation facilitates from the more robust conclusion that one particular study method is superior to another. The latter one needs a more detailed report on how much variability there is between different study runs and which comparisons are still significant after correcting multiplicity.

Questions for authors

How much of the advantage over fixed pipelines remains when the underlying model and total study budget are matched? A controlled comparison on a representative subset of environments would help distinguish improvements from the study strategy from improvements due to additional computation or model choice.

What exactly is resampled in the bootstrap analysis? In particular, do the intervals account for variability across independently generated study artifacts, as well as variability across downstream tasks and solver rollouts? Could you also report which method comparisons remain significant after multiple-comparison correction?

With current workflow execution records, how can we obtain the selector baseline? How are those runs assigned to the selector, and how is their variability represented in the comparison? This would help assess whether the archive-assisted agent gains from selecting a suitable workflow or from executing and combining workflows beyond that selection.

2 comments

We thank the reviewer for identifying our contribution as being well defined and practical. We address their concerns below.

W1. Separating study design from model, budget and workflow selection

Q1. Matching base model and study budget

Addressed in General Response (1/5).

Q3. Archive agent versus selection alone

The construction of the selector baseline is described in Appendix A.2.2. The selector receives a text description of the environment and of each archive workflow, and chooses one workflow without running it or seeing any artifacts. We score each choice with the main-experiment arm for that workflow, so the selector rows contain no new rollouts. The reported SD applies the law of total variance across selections and study runs. Meta-Agent w/ Archive exceeds the selector on five of six benchmarks, by up to 0.145 Avg@3 on DABStep, and the exception is BCP-G, where the selector picks Corpus2Skill. A fixed rule that runs every archive workflow and delivers all their outputs also scores below Meta-Agent w/ Archive on the three benchmarks we tested (response to Reviewer EEAw, W1 Q1), so its gain is not explained by selection alone or by running every workflow.

W2. Confidence in method rankings

Q2. Bootstrapping and multiple-comparison correction

In the submission, the intervals in Tables 6 and 7 resampled tasks only. Each task's score combined its rollouts within each study run, so rollout variability was included, but the three artifact sets were held fixed, so variation across study runs was not. Only Table 8 also resampled study runs.

We reran every comparison with a bootstrap that resamples both tasks and, for each method, its three study runs (within each world for APEX-Agents). Rollouts stay attached to their study run, so their variability still enters through the per-task scores. We apply Holm correction at α=0.05\alpha = 0.05 within each family of comparisons: the 24 primary and 24 transfer comparisons against No Study, and the 6 archive comparisons in Table 8.

After correction, 20 of the 24 primary Avg@3 comparisons remain significant gains over No Study (21 in the submission). After solver transfer, 2 gains and 4 losses remain significant (7 and 5 in the submission). The archive gains on BCP-G and DABStep in Table 8 remain significant. The full procedure and revised tables are in General Response (2/5)–(4/5).

AuthorsOpenReview ↗

Looking Forward To Your Response

Thanks for your review! As the response period is ending soon, we would really appreciate if the reviewer can take a look at our responses and let us know if the concerns are addressed as we have conducted further experiments at their request. Thank you!

Reviewer_EEAwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper formalizes task-agnostic environment preprocessing: before test time and without knowledge of the downstream task distribution, a studying system explores an environment under a budget and produces reusable artifacts (indices, notes, scripts, playbooks) for a frozen solver. The authors compare two open-ended Meta-Agent variants (with and without an archive of executable studying workflows) against two committed task-agnostic baselines (PREPING and CORPUS2SKILL) and a NoStudy baseline across six heterogeneous benchmarks spanning 36 environments. A Meta-Agent variant achieves the highest Avg@3 reward on five of six benchmarks, while fixed corpus processing (CORPUS2SKILL) remains best on the largest corpus benchmark (BCP-G). The paper additionally analyzes how downstream reward scales with study budget, how studied artifacts reduce the test-time sampling needed to reach a given score, and what the studying agents explore and produce.

Critical strengths

1. The paper studies a practical problem with a clear setup.

The paper asks how an agent can prepare for an unfamiliar environment before receiving downstream tasks or evaluation feedback. This is useful when representative task examples are difficult to obtain. The formulation in Sec. 3 provides a common framework for comparing corpus processing, synthetic practice, and open-ended exploration with a fixed downstream solver.

2. The experiments cover different environments and include repeated runs.

The evaluation covers six benchmarks and 36 environments with different corpora and tool interfaces. Each studying method is run independently three times, and each artifact set is also evaluated three times. The authors report study and inference costs separately and clearly state that Best@K assumes oracle selection. These details help assess performance and its variability.

3. The paper reports useful negative results.

The results show that studying is not always helpful. CORPUS2SKILL performs well on BCP-G but reduces performance on some other benchmarks. Meta-Agent w/ Archive performs better than Meta-Agent w/o Archive on BCP-G, but the transfer experiments also show that artifacts useful for one solver may provide limited benefit or even hurt another. These findings help clarify the limitations of environment preprocessing.

Critical weaknesses

1. The source of the gains from open-ended studying is still unclear.

Meta-Agent w/ Archive can explore the environment, execute multiple workflows, inspect their outputs, and combine artifacts. The current experiments do not clearly separate the benefit of adaptive decisions from that of simply running and combining several workflows. Table 8 also shows that, after accounting for variation across study runs, archive access provides significant gains over the variant without the archive only on BCP-G and DABStep. This supports its value in some environments, but does not establish which components are necessary.

Questions:

  1. Under the same budget, could a baseline that runs multiple workflows according to a fixed rule and combines their outputs achieve similar gains? Do the authors have such results, or could they add this comparison on a representative benchmark?
  2. Appendix A.2.2 reports that the selector chooses CORPUS2SKILL in 176 of 180 calls. Do its explanations reflect differences between environments? Could this preference be related to how the environment or workflow descriptions are presented?

2. The lack of cost-controlled comparisons makes the performance gains harder to interpret.

The main results compare methods at their full configurations, but study costs differ substantially even within the same benchmark. For example, on DABStep, adding the archive increases Avg@3 from 0.400 to 0.529, while study cost rises from $0.59 to $19.47. These results show the performance differences between the tested configurations, but it is difficult to tell whether the additional gains come from a better studying strategy or simply more computation.

The reported costs and break-even analysis against NoStudy are useful. However, the latter mainly addresses when study costs can be recovered through reuse. It does not show how different studying methods compare under the same resource constraints.

Questions:

  1. Could the authors compare performance at several shared dollar budgets on one or two representative benchmarks? Matching dollar cost does not fully match computation, but it would still provide a useful comparison for practical deployment.
  2. For a fixed number of downstream tasks NN, how do the methods compare when considering total cost, Ctotal=Cstudy+NCinferenceC_{\mathrm{total}} = C_{\mathrm{study}} + N C_{\mathrm{inference}}, where CinferenceC_{\mathrm{inference}} is the average inference cost per task? At what levels of reuse does the archive-equipped variant offer a better performance–cost trade-off than the variant without the archive?

3. The effects of studying strategies and test-time access mechanisms are not separated.

Appendix A.3.1 shows that the methods differ not only in the artifacts they produce, but also in how the solver accesses and uses information. CORPUS2SKILL disables direct search over the original corpus and requires navigation through its skill hierarchy. Standalone PREPING retrieves playbook entries based on the target instruction, whereas its archive version provides a static playbook. The two Meta-Agent variants also differ in prompt injection and artifact delivery.

Keeping each method’s native configuration is reasonable for comparing complete systems. However, the resulting performance differences reflect both artifact quality and test-time access mechanisms. In particular, some negative results for CORPUS2SKILL may come from the enforced navigation restrictions, rather than from corpus preprocessing itself.

Questions:

  1. Would the results change if the solver could directly search the original corpus while retaining access to the CORPUS2SKILL artifacts, especially on benchmarks where performance decreases?
  2. Could the authors clarify which settings and delivery mechanisms remain the same when a workflow runs independently versus when it is invoked by the Meta-Agent? How might these differences affect the interpretation of outperforming the best single workflow?
Questions for authors

See the weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer EEAw (1/2)

We thank the reviewer for their thoughtful suggestions, for describing our work as a practical problem with a clear setup, and for noting the value of our negative results. We address their questions below.

W1. Source of the gains from open-ended studying

W1 Q1. Fixed-rule multi-workflow baseline under the same budget

We ran the suggested baseline on DABStep, Harvey LAB and OfficeQA. The fixed rule runs all three archive workflows in every study run and gives the solver all of their outputs, with one standard prompt combining each workflow's usage instructions in place of the prefix the Meta-Agent writes. Its mean study cost per run was 18.23 USD on DABStep, 31.59 USD on Harvey LAB and 23.01 USD on OfficeQA, against 19.47, 31.21 and 15.99 USD for Meta-Agent w/ Archive, so it spent about the same on DABStep and Harvey LAB and more on OfficeQA.

Avg@3

MethodDABStepHarvey denseOfficeQA
No Study0.353 ± 0.0100.260 ± 0.0080.549 ± 0.020
Corpus2Skill0.324 ± 0.0090.227 ± 0.0060.535 ± 0.029
PREPING0.464 ± 0.0600.299 ± 0.0130.609 ± 0.020
Fixed rule (all archive workflows)0.305 ± 0.0490.297 ± 0.0170.564 ± 0.022
Meta-Agent w/o Archive0.400 ± 0.0060.361 ± 0.0070.633 ± 0.014
Meta-Agent w/ Archive0.529 ± 0.0970.366 ± 0.0120.615 ± 0.004

Best@3

MethodDABStepHarvey denseOfficeQA
No Study0.487 ± 0.0180.398 ± 0.0130.730 ± 0.021
Corpus2Skill0.446 ± 0.0050.322 ± 0.0100.717 ± 0.010
PREPING0.565 ± 0.0420.426 ± 0.0170.747 ± 0.016
Fixed rule (all archive workflows)0.408 ± 0.0740.437 ± 0.0130.711 ± 0.011
Meta-Agent w/o Archive0.549 ± 0.0120.485 ± 0.0090.780 ± 0.016
Meta-Agent w/ Archive0.653 ± 0.0880.488 ± 0.0080.749 ± 0.006

The fixed rule scores below both Meta-Agent variants on all three benchmarks. It matches PREPING on Harvey LAB, is at the No Study level on OfficeQA, and falls below No Study on DABStep. On DABStep the merged bundle misdirects the solver. Every DABStep instruction asks it to consult the documentation, so 96.8% of trials go through the Corpus2Skill skills and only 11.9% use the PREPING playbook, although the full playbook is in the prompt. Meta-Agent w/ Archive inspected the workflow outputs and dropped Corpus2Skill in all three DABStep study runs. Running and combining every workflow therefore does not reproduce the archive variant's gains, and the DABStep case shows that choosing which outputs to deliver matters.

W1 Q2. Selector choosing CORPUS2SKILL in 176 of 180 calls

The selector's explanations do reflect each environment, for example "can't index 100k docs meaningfully" on BCP-G and "Corpus2Skill adds navigational overhead without answering the domain-reasoning gap" on DABStep. Its case for Corpus2Skill, however, mostly restates the workflow description ("compiles the task-blind corpus into canonical navigational skills"), since it cannot see what the workflow produces. On DABStep, another sample picks Corpus2Skill because "Success hinges on navigating and correctly interpreting these files," yet Corpus2Skill scores below No Study there. The preference persists when we reword the prompt and when we add a count of the environment's files and tools (530 of 540 calls across three reruns), so it does not depend on how the environment is presented, and the workflow descriptions the selector restates are the more likely source. Meta-Agent w/ Archive, which inspects workflow outputs before committing, dropped Corpus2Skill on DABStep in all three runs, where that helped, but also kept no Corpus2Skill output on BCP-G, where Corpus2Skill is the best method. Inspecting outputs changes which workflows are chosen but does not guarantee the right choice.

AuthorsOpenReview ↗

Response to Reviewer EEAw (2/2)

W2. Cost-controlled comparisons

W2 Q1. Performance at shared dollar budgets

Addressed in General Response (1/5).

W2 Q2. Total cost vs. number of downstream tasks, and archive break-even

Based on the reviewer's direct suggestion, we show the Total cost Ctotal=Cstudy+NCinferenceC_{\text{total}} = C_{\text{study}} + N C_{\text{inference}} at each benchmark's test-set size NN, using the paper's mean costs and one rollout per task. Reward change is the lift of Meta-Agent w/ Archive over w/o Archive (Avg@3, Table 8). The last two columns give the number of tasks at which the archive variant reaches lower cost per expected success, Ctotal/(Nr)C_{\text{total}}/(N r), and lower total cost.

BenchmarkNNReward change (pp)Total cost w/o Archive (USD)Total cost w/ Archive (USD)Tasks for lower cost per successTasks for lower total cost
BCP-G830+3.9515.28721.20NeverNever
OfficeQA246−1.961.7778.72NeverNever
Harvey LAB250+0.571.8697.46NeverNever
DABStep450+13.063.1478.87≈359≈2,700
APEX-Agents452+1.0179.06699.92≈180,000Never
AppWorld168−2.436.7944.74≈788≈530

At each benchmark's test-set size, Meta-Agent w/ Archive has the lowest cost per expected success on DABStep, where it overtakes the variant without the archive after about 359 tasks, though lower total cost needs about 2,700 tasks. On the other benchmarks it does not overtake that variant within the test-set size. Across all methods, Meta-Agent w/o Archive is cheapest per success on OfficeQA, Harvey LAB and AppWorld, Corpus2Skill on BCP-G, and No Study on APEX-Agents.

Cost per expected success Ctotal/(Nr)C_{\text{total}}/(N r) in USD at each benchmark's test-set size, lowest per benchmark in bold.

BenchmarkNo StudyPREPINGCorpus2SkillMeta-Agent w/o ArchiveMeta-Agent w/ Archive
BCP-G2.742.371.992.513.04
OfficeQA0.500.510.850.400.52
Harvey LAB0.921.051.380.801.07
DABStep0.350.350.400.350.33
APEX-Agents1.297.791.441.776.65
AppWorld0.340.630.360.320.40

W3. Studying strategies vs. test-time access mechanisms

W3 Q1. CORPUS2SKILL with direct corpus search enabled

Addressed in General Response (5/5).

W3 Q2. Settings and delivery: standalone workflow vs. invoked by the Meta-Agent

Inside the archive, each workflow runs the same study pipeline and models as its standalone version, with settings the controller may change, and the solver, harness and task instruction text are the same. Only delivery differs. PREPING supplies its full playbook instead of retrieving the 40 most relevant entries for each instruction, Corpus2Skill adds its skills without blocking direct corpus search, and the chosen outputs share one system prompt prefix and bundle, whereas the standalone workflows and Meta-Agent w/o Archive add their guidance to the task instruction. Delivery alone does not explain the gains. The fixed rule baseline (W1 Q1) uses this exact delivery for all three workflows and scores below Meta-Agent w/ Archive on all three benchmarks and below No Study on DABStep (0.305 vs. 0.353).

AuthorsOpenReview ↗

Looking Forward To Your Response

Thanks for your review! As the response period is ending soon, we would really appreciate if the reviewer can take a look at our responses and let us know if the concerns are addressed as we have conducted further experiments at their request. Thank you!

Thanks for the additional experiments. The budget-matched comparisons and the new ablations address most of my concerns. While transfer to other solvers remains limited, I find the empirical contribution useful overall. I will raise my score to 3 (weak accept).

Comments on the paper

10 comments
AuthorsOpenReview ↗

General Response (1/5): Cost-Controlled Comparison of Studying Methods

We thank the reviewers for their thoughtful and constructive feedback. We address three common concerns shared by them here.

Cost-Controlled Comparison of Studying Methods

All reviewers note that the studying methods consume very different resources, so the comparisons in the paper do not separate studying strategy from the amount spent on it (EEAw W2, C8ox Q5, tkqA W1 and Q1). We agree, and report dollar-matched runs below.

We would like to caveat, however, that matching resources across these methods is difficult because each one spends its budget in a different way and, as Reviewer EEAw notes, equal dollar cost does not imply equal computation. PREPING spends its budget on cycles of synthetic practice, and the Meta-Agent variants on agent turns of exploration and artifact writing. Corpus2Skill processes every document in the corpus with LLM generators and embedding models, so its cost is set by corpus size rather than by a budget. Capping it would mean processing only part of the corpus, which would no longer be true to the original method, so we exclude Corpus2Skill from this comparison. We emphasize that dollar cost here is a practical proxy for resources consumed.

We ran PREPING, Meta-Agent w/o Archive and Meta-Agent w/ Archive with fixed study budgets of USD 1, 5, 10 and 25 on DABStep, OfficeQA and AppWorld. Every study run spent its full budget. For PREPING we set the maximum number of cycles high enough that the budget, not the cycle limit, ended the run. For the Meta-Agent variants, we continued the agent loop with a message stating how much of the dollar budget remained. Each cell is the mean ± sample SD over three independent study runs, each evaluated with three Haiku 4.5 repetitions over all tasks.

At every matched budget, a Meta-Agent variant has the highest Avg@3 on all three benchmarks except DABStep at USD 1, so the advantage over PREPING does not come from spending more. Additional budget helps Meta-Agent w/ Archive, which rises from 0.386 to 0.580 on DABStep and from 0.628 to 0.731 on AppWorld between USD 1 and USD 25, while the variant without the archive changes by less than 0.03. On OfficeQA all three methods are within 0.023 of each other at every budget.

Avg@3

MethodBudget (USD)DABStepOfficeQAAppWorld
PREPING10.419 ± 0.0050.578 ± 0.0500.665 ± 0.016
PREPING50.369 ± 0.0430.611 ± 0.0380.624 ± 0.036
PREPING100.341 ± 0.0140.590 ± 0.0390.657 ± 0.049
PREPING250.350 ± 0.0400.599 ± 0.0140.628 ± 0.021
Meta-Agent w/o Archive10.408 ± 0.0030.586 ± 0.0280.677 ± 0.016
Meta-Agent w/o Archive50.418 ± 0.0210.615 ± 0.0130.665 ± 0.024
Meta-Agent w/o Archive100.405 ± 0.0020.599 ± 0.0370.685 ± 0.016
Meta-Agent w/o Archive250.413 ± 0.0040.599 ± 0.0400.691 ± 0.053
Meta-Agent w/ Archive10.386 ± 0.0240.601 ± 0.0240.628 ± 0.018
Meta-Agent w/ Archive50.466 ± 0.1270.613 ± 0.0310.678 ± 0.042
Meta-Agent w/ Archive100.562 ± 0.1210.592 ± 0.0230.676 ± 0.058
Meta-Agent w/ Archive250.580 ± 0.0230.617 ± 0.0230.731 ± 0.044

Best@3

MethodBudget (USD)DABStepOfficeQAAppWorld
PREPING10.551 ± 0.0170.755 ± 0.0410.776 ± 0.009
PREPING50.478 ± 0.0620.763 ± 0.0230.744 ± 0.024
PREPING100.451 ± 0.0430.726 ± 0.0380.762 ± 0.042
PREPING250.472 ± 0.0550.756 ± 0.0220.750 ± 0.018
Meta-Agent w/o Archive10.555 ± 0.0070.732 ± 0.0250.786 ± 0.016
Meta-Agent w/o Archive50.586 ± 0.0380.752 ± 0.0040.776 ± 0.046
Meta-Agent w/o Archive100.545 ± 0.0080.748 ± 0.0210.792 ± 0.026
Meta-Agent w/o Archive250.573 ± 0.0130.749 ± 0.0060.798 ± 0.022
Meta-Agent w/ Archive10.511 ± 0.0240.743 ± 0.0190.728 ± 0.018
Meta-Agent w/ Archive50.602 ± 0.1030.766 ± 0.0190.780 ± 0.026
Meta-Agent w/ Archive100.672 ± 0.0970.748 ± 0.0110.786 ± 0.036
Meta-Agent w/ Archive250.694 ± 0.0380.749 ± 0.0220.827 ± 0.045
AuthorsOpenReview ↗

General Response (2/5): Statistical Tests Across Study Runs

Reviewers C8ox (W2, Q2) and tkqA (W2, Q2) note that the significance tests in the submission resample tasks only, so they do not reflect variation across independent study runs, and that they are not corrected for multiple comparisons. We reran every comparison with a bootstrap that also resamples study runs, and applied a multiple-comparison correction.

Original procedure. In the submission, significance came from a paired task bootstrap. For each benchmark we computed one score per task for each method and the matching No Study score. Avg@3 averages all nine rollouts of a task (three study runs × three repetitions). Best@3 takes the maximum over the three repetitions within each study run and averages over study runs. No Study uses ten repetitions in the primary experiment and three after transfer. We resampled tasks with replacement 50,000 times and marked a difference when the uncorrected 95% percentile interval excluded zero. Rollout variability entered through each task's score, but each method's three artifact sets were held fixed, so the intervals in Tables 6 and 7 did not reflect variation across study runs. Only Table 8 also resampled study runs.

Revised procedure. Each bootstrap replicate now resamples the tasks with replacement and, independently for each method, resamples its three study runs with replacement. The task draw is shared by the methods being compared, so differences stay paired. No Study has no study runs, so only its tasks are resampled. APEX-Agents consists of 31 separately studied worlds, so its study runs and tasks are resampled within each world and the worlds are combined with their original task weights. Rollouts are not resampled separately. Each task's score within a study run keeps its observed rollouts (their mean for Avg@3 and their maximum for Best@3), so rollout variability enters through those scores, as in the original procedure. We use 1,000,000 replicates and report the 95% percentile interval of the paired difference. The two-sided p value comes from the null-centered bootstrap distribution, p=1+N1+Bp = \frac{1 + N}{1 + B}, where NN is the number of replicates with ∣d∗−d∣≥∣d∣\lvert d^{\ast} - d \rvert \ge \lvert d \rvert, dd is the observed difference, d∗d^{\ast} a replicate, and BB the number of replicates. Holm correction at α=0.05\alpha = 0.05 is applied within each family of comparisons, separately for Avg@3 and Best@3. The families are the 24 primary comparisons against No Study, the 24 transfer comparisons against No Study, and the 6 archive comparisons in Table 8. Intervals below are uncorrected 95% intervals; significance is decided by the Holm-adjusted p value.

Results. After study-run resampling and Holm correction, 20 of the 24 primary Avg@3 comparisons remain significant gains over No Study (21 in the submission). The gain that is no longer significant is Corpus2Skill on AppWorld. The Corpus2Skill losses on Harvey LAB and DABStep remain significant. After solver transfer, 2 Avg@3 gains and 4 Avg@3 losses remain significant (7 and 5 in the submission). The archive gains on BCP-G and DABStep in Table 8 remain significant.

Table 8 (revised). Meta-Agent w/ Archive minus Meta-Agent w/o Archive.

BenchmarkAvg@3 diffAvg@3 95% CIAvg@3 Holm pBest@3 diffBest@3 95% CIBest@3 Holm p
BCP-G+0.039[+0.012, +0.069]0.034+0.041[+0.005, +0.078]0.149
OfficeQA-0.019[-0.048, +0.011]0.843-0.031[-0.068, +0.005]0.411
Harvey LAB+0.005[-0.017, +0.029]1.000+0.003[-0.025, +0.032]1.000
DABStep+0.130[+0.034, +0.220]0.004+0.104[+0.025, +0.196]0.098
APEX-Agents+0.010[-0.006, +0.026]0.843+0.014[-0.010, +0.039]0.764
AppWorld-0.024[-0.097, +0.052]1.000-0.002[-0.077, +0.077]1.000
AuthorsOpenReview ↗

General Response (3/5): Revised Table 7, Paired Differences from No Study

Table 7 (revised). Paired differences from No Study, primary experiment.

BenchmarkMethodAvg@3 diffAvg@3 95% CIAvg@3 Holm pBest@3 diffBest@3 95% CIBest@3 Holm p
BCP-GPREPING+0.088[+0.069, +0.105]< 0.001+0.092[+0.067, +0.118]< 0.001
BCP-GCorpus2Skill+0.273[+0.246, +0.301]< 0.001+0.329[+0.295, +0.364]< 0.001
BCP-GMeta-Agent w/o Archive+0.050[+0.029, +0.069]< 0.001+0.060[+0.031, +0.088]0.002
BCP-GMeta-Agent w/ Archive+0.089[+0.067, +0.113]< 0.001+0.101[+0.072, +0.132]< 0.001
OfficeQAPREPING+0.060[+0.027, +0.092]0.004+0.017[-0.019, +0.052]0.717
OfficeQACorpus2Skill-0.014[-0.053, +0.024]0.469-0.013[-0.045, +0.020]0.717
OfficeQAMeta-Agent w/o Archive+0.084[+0.055, +0.113]< 0.001+0.051[+0.017, +0.085]0.044
OfficeQAMeta-Agent w/ Archive+0.066[+0.040, +0.092]< 0.001+0.020[-0.009, +0.049]0.717
Harvey LABPREPING+0.039[+0.019, +0.059]0.002+0.028[+0.001, +0.055]0.208
Harvey LABCorpus2Skill-0.033[-0.055, -0.012]0.016-0.076[-0.106, -0.046]< 0.001
Harvey LABMeta-Agent w/o Archive+0.100[+0.078, +0.123]< 0.001+0.087[+0.059, +0.116]< 0.001
Harvey LABMeta-Agent w/ Archive+0.106[+0.082, +0.130]< 0.001+0.090[+0.062, +0.118]< 0.001
DABStepPREPING+0.111[+0.054, +0.172]< 0.001+0.078[+0.033, +0.126]0.014
DABStepCorpus2Skill-0.029[-0.048, -0.010]0.016-0.041[-0.067, -0.016]0.023
DABStepMeta-Agent w/o Archive+0.047[+0.030, +0.064]< 0.001+0.062[+0.033, +0.092]0.002
DABStepMeta-Agent w/ Archive+0.176[+0.081, +0.267]< 0.001+0.166[+0.089, +0.256]< 0.001
APEX-AgentsPREPING+0.047[+0.032, +0.062]< 0.001+0.045[+0.024, +0.067]< 0.001
APEX-AgentsCorpus2Skill+0.037[+0.020, +0.055]< 0.001+0.034[+0.009, +0.059]0.057
APEX-AgentsMeta-Agent w/o Archive+0.040[+0.024, +0.057]< 0.001+0.032[+0.009, +0.055]0.057
APEX-AgentsMeta-Agent w/ Archive+0.050[+0.033, +0.067]< 0.001+0.046[+0.023, +0.070]0.002
AppWorldPREPING+0.069[+0.014, +0.123]0.043+0.075[+0.015, +0.135]0.093
AppWorldCorpus2Skill+0.027[-0.011, +0.067]0.345+0.025[-0.015, +0.065]0.717
AppWorldMeta-Agent w/o Archive+0.112[+0.051, +0.178]0.003+0.094[+0.027, +0.164]0.057
AppWorldMeta-Agent w/ Archive+0.088[+0.040, +0.143]0.007+0.092[+0.043, +0.149]0.017

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.