- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces DAEDALUS, a framework for constructing reusable procedural memory before deployment, without curated task instructions or an oracle verifier during memory generation. An Explorer proposes and executes synthetic tasks, while a Solver attempts them in reset environments. Following failures, an Extractor produces or revises textual heuristics. A heuristic is retained after the Solver achieves three consecutive judged successes. Solver outcomes also guide task-difficulty refinement. Accepted heuristics are consolidated into a frozen memory bank for downstream use. Experiments cover AppWorld’s normal split, the retail domain of τ²-bench, and the Operations split of AutomationBench. The improvement validates the efficiency of the proposed methods.
Critical strengths
-
The presentation and writing are good and easy to follow.
-
The investigating problem that constructing reusable procedural memory is very important.
-
Table 1 shows improvements over no memory across three different environments, including gains in repeated run reliability, which strengthens the methods' credibility.
-
The reported deterioration at larger generation budgets is also informative and appropriately limits the paper’s scaling claims.
Critical weaknesses
-
DAEDALUS uses substantially more solver rollouts than most baselines, leaving its advantage under comparable computation unclear. So could the authors provide the computation cost comparison?
-
The ablations change retries, heuristic revision, and validation together, making the specific benefit of repeated-success filtering unclear.
Questions for authors
-
What are PREPING’s total memory construction cost and environment interaction budget? Under comparable construction budgets, does DAEDALUS retain its advantage?
-
How much do the main AppWorld result and the solver loop ablation vary across independently generated and consolidated banks?
3 comments
Response to the Rebuttal
Thanks for your careful rebuttal. My concerns are addressed and I am willing to maintain my positive score to support this paper to be accepted.
We thank the reviewer again for their support throughout the discussion. Their questions on cost and comparable budgets led to the generation-cost comparison of Appendix F, which clarifies the trade-offs of our method.
We thank the reviewer for their feedback, and for appreciating the clarity of the paper, the importance of reusable procedural memory, the consistent gains across three environments, and the scaling experiment. We address each concern below. All changes in the revised paper are marked in red.
W1. Computation cost
We agree that the comparison needs the cost of every method, which the revision now reports, together with solver rollouts (Appendix F). DAEDALUS costs more to generate than the baselines: 142 USD on AppWorld for 1,500 judged solver rollouts, against at most 18 USD for the baselines. This cost is paid once, however, whereas inference is paid on every task. DAEDALUS is the cheapest method at inference on AppWorld and τ², as cheap as the no-memory agent (Table 1), so its total cost per solved task falls below that of every baseline after enough task runs (Table 16).
Changes to the paper. Section 4.2; Appendix F, Tables 15–16 (costs, rollouts, break-even).
W2. The ablation changes three factors together
We agree that row D of Table 3 does not isolate repeated-success filtering. These components are hard to separate, because they only work together: a heuristic can only be revised after a failed retry, and only validated through further retries. We therefore evaluate the Solver loop as a whole.
Changes to the paper. Section 5.1 now attributes the gain to the Solver loop as a whole.
Q1. PREPING's cost and comparison at equal budget
PREPING runs 100 practice tasks, for at most 16 USD on AppWorld. To compare the two methods at equal cost, we stop DAEDALUS's generation on AppWorld once it reaches PREPING's budget, and consolidate only the heuristics accepted up to that point, for each of three banks. DAEDALUS then matches PREPING with a higher mean (51.1 ± 1.1 vs. 47.1 ± 5.3 MSR) and five times less variation across banks.
Changes to the paper. Appendix F (comparison at equal budget).
Q2. Variance across independently generated banks
We now build 3 independent memory banks for every method and benchmark, and Table 1 reports the mean ± SE across banks. On AppWorld, the main result of DAEDALUS is 56.8 ± 2.7 MSR across banks. DAEDALUS still scores above PREPING on AppWorld and τ² (56.8 ± 2.7 vs. 47.1 ± 5.3, and 70.7 ± 2.0 vs. 58.8 ± 1.4), and matches it on AutomationBench (34.6 ± 1.6 vs. 32.7 ± 1.1). Due to budget constraints, we could not also re-run the full ablation across banks, so Table 3 still reports one bank per row.
Changes to the paper. Table 1 and Section 4.1.
We thank the reviewer again for their positive assessment and useful feedback. We hope that the cost comparison, the comparison at equal budget and the results over three banks address their remaining concerns.