acceptodds
Under review as a conference paper at ICLR 2027

Unsupervised Tool Evolution for Coding Agents in Data-Lake Analytics

Abstract

Coding agents can answer analytical questions over a data lake by discovering, cleaning, and joining the relevant data. However, they discard this work after each question and redo it from scratch for the next one. Tool evolution, which automatically builds and refines reusable tools across tasks, is a natural fit for this repetition. Since a ground-truth label for a data-lake question requires an expert to solve it first, we consider the unsupervised setting. We propose TEAL (Tool Evolution for Analytics over data Lakes), which evolves a library of tools and usage skills for a given Worker agent without labels. Two agents propose tools from complementary evidence, an Explorer from the data lake itself and a Critic from Worker trajectories. A proposed change is kept only if the Worker adopts the tool on a rerun without extra steps and any changed answer is supported by trajectory evidence. We also introduce TEAL-Q, which generates synthetic questions for tool evolution when real ones are unavailable. We evaluate on KramaBench and LakeQA with four Worker LLMs under step budgets, which cap the number of steps the Worker may take per question. With a tight budget of 10 or 15 steps, TEAL improves every Worker on both benchmarks, lifting Sonnet-5 at 15 steps by 9.3 points on KramaBench and 21.7 points on LakeQA. A tighter budget also yields better feedback for evolution. For example, Sonnet-5’s LakeQA library evolved at 20 steps and deployed unchanged at 30 steps scores 68.3, above both the baseline (59.6) and a library evolved directly at 30 steps (64.2). Additionally, a library evolved on synthetic questions performs comparably to one evolved on real questions. These results show that useful tools can be evolved without labels, and we hope they inspire future work on what to evolve and how to obtain feedback in unsupervised agent evolution.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Tool Evolution, LLM Agents, Self-Evolving Agents, Agentic Data Analytics, Coding Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oo4EOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses an important yet underexplored limitation of current coding agents in data lake settings: each task is tackled from scratch, so knowledge and artifacts from previous work are discarded rather than reused. To address this, the authors propose TEAL, a framework for tool evolution in data lake analytics that progressively builds and refines a library of reusable tools together with useful skills distilled from past tasks. TEAL is a multi-agent system in which agents take on distinct, specialized roles. The authors further introduce TEAL-Q, a component that generates synthetic questions to drive and support tool evolution. Experiments on KramaBench and LakeQA show that the proposed approach is effective.

Critical strengths
  1. The paper is well organized and easy to follow. The illustrations are vivid and easy to understand.
  2. The proposed method is simple, intuitive, and easy to understand. The experimental results demonstrate a certain degree of effectiveness.
Critical weaknesses
  1. The method is relative simple and its multi-agent design (TEAL) lacks a clearly novel technical component, limiting the incremental contribution.
  2. Library-evolution variants degrade notably between 20 and 30 steps, in some cases severely; this contradicts the paper's main claim and is left unexplained — a theoretical account and finer-grained experiments are needed.
  3. Ablations isolating each agent's role and contribution are missing, leaving the empirical support incomplete.
Questions for authors

N/A

3 comments
AuthorsOpenReview ↗

Response to Reviewer oo4E [1/1]

Q1: Novelty

Thank you for this comment. We believe TEAL provides a new tool evolution system for data lake analytics, with three contributions.

  1. Persistent data artifacts. Through the Explorer, TEAL builds tools backed by persistent data artifacts, such as catalogs and indexes over the data lake, which are central to analytics over data lakes. Across all 20 LakeQA libraries (four Workers, five budgets), Explorer-created tools receive 87% of the Worker's library calls at test time. In Appendix A, the Sonnet-5 Worker calls the catalog tool search_dataset_catalog in all 80 LakeQA test tasks. Without the Explorer, Gemini-3.1-Pro's accuracy on LakeQA at k=15 drops from 50.83 to 45.83.
  2. Unsupervised evolution, from question generation to the library. Prior unsupervised methods accept a tool if it executes (ISE) or a skill if sampled answers agree (DataCOPE). TEAL accepts a change only if the Worker uses it on a rerun, finishes within m extra steps, and any change in the answer is supported by the trajectory evidence. With the same Sonnet-5 Worker and Opus-4.8 evolution model, ISE falls 10.0 points below the baseline on LakeQA at k=15, while TEAL gains 21.7 points (Table 1). TEAL-Q generates the questions from the data lake, and libraries evolved on them match those evolved on real questions for all four Workers (Tables 2 and 7).
  3. Findings on the step budget. Libraries evolved under a tight step budget transfer unchanged to looser budgets, and the evolution budget can be chosen without labels from the steps the library saves on unlabeled questions. We hope this inspires future work on how to obtain feedback in unsupervised agent evolution.

Q2: Drops below the baseline at 20 to 30 steps

Our main claim is about evolving under a tight budget and deploying at any looser budget (Sec. 4.2). Every library that falls below the baseline in Fig. 3 was evolved at 20 steps or more, and libraries evolved at a tight budget stay above the baseline at every looser budget, for every Worker on both benchmarks.

The severe drops (5 points or more) occur only for Gemini-3.8-Flash. Our further analysis shows that this Worker differs from the other three as follows.

  • Extra steps add little. Its 10-step baseline already reaches 87% of its 30-step accuracy (30% to 49% for the other Workers), with a third of the steps (8.9 vs. 27.0).
  • It often runs out of steps without writing an answer, despite the instruction to answer in the final steps (L318-319). In the LakeQA k=30 baseline, 13% of answers are empty, vs. at most 4% for every other Worker in every cell.
  • Its looser-budget library prescribes a longer workflow, e.g., on LakeQA, eight stages at k=30 vs. two tool calls at k=15 (search_catalog, then query_files). This coincides with more empty answers (28% vs. 13% for the k=30 baseline).

Q3: Per-agent ablations

Thank you for the suggestion. Sec. 4.5 ablates the Explorer, the Skill Writer, and the label-free verification, with Sonnet-5 as the Worker at k=15 on KramaBench. We add an Explorer ablation on LakeQA and an audit of the Analyst.

  • Explorer. Removing the Explorer lowers accuracy on both benchmarks. On KramaBench it drops from 72.54 to 71.59. On LakeQA's larger data lake the drop is larger, from 50.83 to 45.83 for Gemini-3.1-Pro at k=15. Across the 20 LakeQA libraries, Explorer-created tools receive 87% of the Worker's library calls at test time. Without the Explorer, evolution no longer produces the artifact-backed tools shown in Appendix A.
  • Skill Writer. Removing the evolved skills costs 3.85 points, because the Worker no longer knows how to use the tools. Tools and their usage skills must be evolved together.
  • Verification and the Analyst. Giving TEAL the ground-truth answers during evolution gives similar accuracy (72.73 vs. 72.54), so the label-free checks already obtain the feedback that gold answers would provide. ISE, which keeps every tool that executes, is 5.7 points below TEAL on KramaBench and 10.0 points below the baseline on LakeQA with the same Worker and evolution model (Table 1). We also audited the Analyst post hoc against the gold answers of the training questions, which TEAL never sees, over its 295 judgments in the 40 evolution runs. When exactly one of the two answers is correct, the Analyst favors it in 86 of 101 cases (85%), and a change whose rerun answer is wrong while the original was right passes in only 11 of 219 cases (5.0%).
  • Critic and Toolsmith. The Critic proposes tool changes from the Worker's trajectories, and the Toolsmith implements them, so removing either one leaves nothing to evolve. The Critic channel alone already gives most of TEAL's gain on KramaBench (+8.36 of +9.31, the "- Explorer" row).

Thank you for your time and effort in reviewing our paper. We have tried to address each of your concerns with clarifications and new results, and we would be grateful if you could take them into account.

Dear Reviewer, The authors have posted their rebuttal responses. I would kindly encourage you to review the authors' responses and continue the discussion by providing folow-up comments where appropriate. In particular please indicate whether the authors responses have adequately addressed your concerns and update your assessment if necessary Thank you for your time and continued efforts! Your AC

AuthorsOpenReview ↗

Looking forward to your feedback

Dear Reviewer oo4E,

Thank you again for your time and thoughtful review. As the discussion period ends in a few hours, we would like to check whether our responses address your concerns. In particular, we clarified our main claim and added an analysis of why the severe drops occur only for Gemini-3.8-Flash (Q2). For per-agent ablations, we added an Explorer ablation on LakeQA and a gold-answer audit of the Analyst checks (Q3). We would be glad to clarify anything further before the deadline.

Best regards,

Authors of Submission46103

Reviewer_FLcwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes TEAL, which evolves a library of tools and usage skills for datalake analytics, without labels. The tools are proposed by two independent agents, an explorer which analyzes the datalake, and a critic agent that looks at rollouts generated by the worker agent. The proposed change is kept if the worker adopts the tool on a rerun without extra steps and if any change in the answer is supported by trajectory evidence. The paper also proposes another variant TEAL-Q that generates synthetic questions for tool evolution, when human questions are scarce. Experimentally they report, that under tight budgets, TEAL improves on two benchmarks (KramaBench and LakeQA).

Critical strengths
  • The paper is well written and the unsupervised tool evolution setting is well motivated.
  • The finding that tools evolved under tighter budgets transfer to looser budgets and outperform them, is interesting.
Critical weaknesses
  • The main issue with the paper is that the verification does not check for correctness. The Analyst treats the final value appearing in the tool output as evidence (Sec 3.2.2), but this need not imply correctness. The false positive rate of their verification is not analyzed. Additionally answer correctness is not verified when the answer is unchanged. More importantly, "TEAL keeps a change if at least one rerun passes all three checks and reverts it otherwise". Its not made clear or motivated or ablated as to why this is the case, even if it can lead to degradation of other answers.
  • Another major gripe with this paper is that the step budget is not a fair and a invariant measure of compute. Tools fold a lot of functionality and compute into a single step whereas an agent without tools would need more steps to be able to achieve the same end result. The tool cap essentially handicaps agents without tools. Moreover, the papers states that step budgets bound cost and latency, but this may not be the case, when each step is itself a complex sequence of operations packaged into 1 tool call. Matched cost comparisons are missing.
  • The paper targets a niche setting of datalake analytics and only compares on 2 benchmarks with small test sets. Furthermore, TEAL's gains shrink as step budgets loosen, and sec 4.2 shows that libraries evolved at looser budgets can fall below the baseline.
  • Novelty is limited. As the paper itself states, unsupervised tool evolution (ISE) and unsupervised skill discovery for data analytics (DataCOPE) already exist. TEAL's main differences are the explorer agent and re-run based heuristic acceptance checks are more limited that what the paper would suggest. Table 3 shows that removing the explorer agent changes the accuracy within noise.
  • The acceptance checks to accept proposed tools are heuristics not related to correctness as mentioned earlier. These are not ablated as to whats critical and whats not. Also, the evolution model is set to Opus 4.8 and not ablated. The gain can come from the stronger model's ability and it needs to be shown that evolution procedure is the critical bit here.
  • TEAL-Q's claim that its questions are well posed and answerable is not verified. The pipeline only checks if the answer is reproducible via the pipeline, not that the answer is correct or the question has a unique interpretation. No human study of these generated questions is performed.
Questions for authors

See Weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer FLcw [1/3]

Q1: Reliability of verification without labels

Thank you for raising this concern.

The main issue with the paper is that the verification does not check for correctness. The Analyst... but this need not imply correctness. The false positive rate of their verification is not analyzed.

Since no label-free verifier can certify correctness, we measured how often ours is wrong. We audited the Analyst (check 3) post hoc over its 295 judgments in the 40 evolution runs, against the gold answers of the training questions, which TEAL never sees. Its false-positive rate is 5.0%, since only 11 of the 219 changes it passes turn a correct original answer into a wrong one. When exactly one of the two answers is correct, the Analyst favors it in 85% of cases, and in 88% of inconclusive cases both answers are wrong.

Additionally answer correctness is not verified when the answer is unchanged.

When the answer does not change, the change cannot alter correctness on that question. Such a change is kept only if the tool is used and the rerun takes at most m extra steps.

More importantly, "TEAL keeps a change...". Its not made clear or motivated or ablated as to why this is the case, even if it can lead to degradation of other answers.

We keep a change if at least one rerun passes, for two reasons. First, the Worker is stochastic. Two reruns of the same question with the same library give different answers in 34% of pairs (345 of 1,010). Second, a wrong answer usually comes from several tools and steps together, not from the changed tool alone. A failed rerun therefore does not necessarily show that the change is harmful, and requiring every rerun to pass would reject useful changes because of noise or errors elsewhere in the pipeline.

(From Q5) The acceptance checks to accept proposed tools are heuristics not related to correctness as mentioned earlier. These are not ablated as to whats critical and whats not.

The three checks serve different purposes, and check 1 is a precondition for the other two, so they cannot be removed independently. Check 1 requires the Worker to call the changed tool on the question that motivated the change. If the Worker does not use it even there, nothing guarantees that it will be used later, and checks 2 and 3 cannot judge its effect. Check 2 requires the rerun to still finish within m extra steps, so a change does not cost the Worker its step budget. Check 3 requires any change in the answer to be better supported by the trajectory, and it is the only check on the answer itself.

We therefore ablate the verification stage as a whole. Replacing it with a gold-answer oracle gives similar accuracy (72.73 vs. 72.54, Table 3), and ISE, which keeps every tool that executes, falls below TEAL (Table 1).

Q2: Matched-cost comparison

Thank you for raising this concern.

...the step budget is not a fair and a invariant measure of compute... Moreover, the papers states that step budgets bound cost and latency, but this may not be the case...

We apologize for the misunderstanding caused by our wording. We use a step budget to constrain each agent, not as a measure of compute to compare across agents. A step budget is common for agents [1], since a token budget is hard to set in advance and often not followed [2]. Since a TEAL step can do more work than a baseline step, we compare compute across methods by cost (below). In the revision, we will replace the unsupported claim that step budgets bound cost and latency with the motivation above, citing [1] and [2].

Matched cost comparisons are missing.

Appendix C reports accuracy against the steps actually used (Fig. 4) and against the estimated cost per episode (Fig. 5). For Sonnet-5, Gemini-3.1-Pro, and Gemini-3.8-Flash, TEAL reaches higher accuracy at the same cost, or the same accuracy at lower cost. For Opus-4.8, whose input tokens are the most expensive, the cost advantage of TEAL is limited, but TEAL reaches accuracy that the baseline does not reach at any budget we tested. A larger budget is unlikely to close this gap, since at k=30 the baseline uses on average only about 10 (KramaBench) and 17 (LakeQA) of its 30 steps (Fig. 4).


[1] Budget-Aware Tool Use Enables Effective Agent Scaling. arXiv:2511.17006.

[2] Token-Budget-Aware LLM Reasoning. arXiv:2412.18547.

AuthorsOpenReview ↗

Response to Reviewer FLcw [2/3]

Q3: Scope, test-set size, and gains at loose budgets

The paper targets a niche setting of datalake analytics...

Analytics over data lakes is a realistic application. Few benchmarks pose questions over a whole data lake, because labeling even one question requires an expert to discover, clean, and join the data first (Sec. 1). KramaBench and LakeQA differ in scale and file types (1,742 mixed files vs. 4,180 tables), and we evaluate four Workers, five budgets, three test repeats, and four harnesses.

...and only compares on 2 benchmarks with small test sets.

To address the test-set size, we compare TEAL with the baseline task by task at the same budget, with each averaged over its three repeats. A single Worker and budget has limited power on 53 or 80 tasks, so we average the per-task gains over several Workers and budgets and test whether the average is above zero (two-sided sign-flip permutation test, with 95% bootstrap intervals over tasks). The gains are significant on both benchmarks.

Gain over the baseline (points)KramaBench (53 tasks)LakeQA (80 tasks)
Evolved and tested at k=15, four Workers+6.9 [+3.0, +10.9], p = 0.001+14.5 [+10.1, +19.0], p < 0.0001
Frozen libraries in Fig. 3, tested at looser budgets only+4.1 [+1.3, +7.1], p = 0.006+9.0 [+6.1, +12.0], p < 0.0001

Furthermore, TEAL's gains shrink as step budgets loosen, and sec 4.2 shows that libraries evolved at looser budgets can fall below the baseline.

Every library that falls below the baseline in Fig. 3 was evolved at 20 steps or more. We do not recommend evolving at a loose budget, because the Worker is no longer pressed for steps and its trajectories give weak feedback (Sec. 4.2). A library evolved at a tight budget stays above the baseline at every looser budget, and its average gain over the looser budgets it is deployed at is significant on both benchmarks (second row of the table above).

Q4: Novelty and the Explorer ablation

Novelty is limited. As the paper itself states, unsupervised tool evolution (ISE) and unsupervised skill discovery for data analytics (DataCOPE) already exist...

TEAL differs from both prior methods in what it evolves, where proposals come from, and how changes are accepted, as summarized below.

What is evolvedSource of proposalsAcceptance without labels
ISEcallable toolsthe Worker's own task needsthe tool executes without error
DataCOPEprocedural skills (text)sampled trajectoriesagreement among sampled answers
TEALcallable tools, persistent data artifacts (catalogs, indexes), usage skillsthe data lake itself (Explorer) and Worker trajectories (Critic)the Worker's behavior on a rerun (use, completion, evidence for a changed answer)

Compared with ISE, TEAL's acceptance rule matters in practice, since a tool that executes does not necessarily help the Worker. In Table 1, with the same Sonnet-5 Worker, the same Opus-4.8 evolution model, and the same 15-step budget, ISE falls 5.7 points below TEAL on KramaBench, and on LakeQA it falls 10.0 points below the baseline, while TEAL gains 21.7 points.

Compared with DataCOPE, which evolves only skills, TEAL evolves callable tools. With skills alone, the Worker still carries out every discovery and cleaning step itself, whereas a single tool call replaces a multi-step exploration. Some tools are also backed by persistent artifacts, such as catalogs and indexes, that are built once over the data lake and reused across questions. On LakeQA, these artifact-backed tools carry most of the Worker's work, as we show for the Explorer below.

We also contribute findings on the step budget. Libraries evolved under a tight budget transfer unchanged to looser budgets, and the evolution budget can be chosen without labels, as a tight budget at which the library saves steps on unlabeled questions. This can inspire future work on how to obtain feedback in unsupervised agent evolution.

AuthorsOpenReview ↗

Response to Reviewer FLcw [3/3]

Table 3 shows that removing the explorer agent changes the accuracy within noise.

We agree that removing the Explorer lowers KramaBench accuracy only slightly (72.54 to 71.59, Table 3). We therefore also ran the ablation on LakeQA. Sonnet-5, the Worker in Table 3, is currently unavailable to us, so we use Gemini-3.1-Pro as the Worker at k=15 and compare with its TEAL result. Without the Explorer, accuracy drops from 50.83 ± 0.72 to 45.83 ± 1.44.

The gap between the two benchmarks comes from their data lakes. On KramaBench's smaller data lake, Explorer-created tools receive only 18.5% of the Worker's library calls, and most discovery goes through tools proposed by the Critic. On LakeQA's larger data lake, across the 20 libraries (four Workers, five budgets), Explorer-created tools make up about half of the final tools (85 of 167) but receive 87% of the Worker's library calls at test time, and 98% of test episodes call at least one of them. For example (Appendix A), the Explorer builds search_dataset_catalog, a ranked search over a catalog of the whole data lake that is built once and reused by every later question. The Sonnet-5 Worker calls it in all 80 LakeQA test tasks. On the Albany County crime-rate task, every TEAL repeat finishes within four steps through this catalog, while the baseline spends nine to ten steps on find and grep.

Q5: Dependence on the Opus-4.8 evolution model

Also, the evolution model is set to Opus 4.8 and not ablated. The gain can come from the stronger model's ability and it needs to be shown that evolution procedure is the critical bit here.

Like other agent and harness optimization methods, TEAL relies on a capable evolution model. [3] makes the same observation, that harness optimization only became practical after recent improvements in coding agents. However, three observations show that the gains come from the procedure rather than from a stronger model alone.

  • ISE uses the same Opus-4.8 evolution model and the same Worker, yet falls 5.7 points below TEAL on KramaBench and 10.0 points below the baseline on LakeQA (Table 1).
  • With Opus-4.8 as the Worker, the evolution model is no stronger than the Worker, and TEAL still gains 5.1 (KramaBench) and 5.0 (LakeQA) points at k=15 (Table 7).
  • In an early experiment in which Sonnet-5 runs every role, both Worker and evolution, TEAL gains 8.7 points on KramaBench at k=15, so evolution does not require the strongest or most expensive model.

[3] Meta-Harness: End-to-End Optimization of Model Harnesses. arXiv:2603.28052.

Q6: Quality of TEAL-Q questions

TEAL-Q's claim that its questions are well posed and answerable is not verified. The pipeline only checks if the answer is reproducible via the pipeline, not that the answer is correct or the question has a unique interpretation.

TEAL uses only TEAL-Q's questions, never its answers, so the correctness of a synthetic answer does not enter evolution. What matters is whether the questions lead evolution to tools that help on real questions, and Sec. 4.4 tests exactly that. For all four Workers on both benchmarks, the library evolved on 50 synthetic questions improves over the baseline by 3.3 to 23.8 points on the real held-out tests, and it stays within 3 points of the library evolved on real questions in every cell. We agree that "well posed" overstates what the pipeline check guarantees, and we will reword it to "answerable by an executable pipeline over the data lake".

No human study of these generated questions is performed.

We audited a random sample of 25 of the 50 synthetic questions per benchmark. For each question, we checked every factual premise it states against the data lake and solved it independently from the raw files.

  • Answerable. All 50 can be answered from the data lake, consistent with TEAL-Q's pipeline check, and none rests on a false premise.
  • Single answer. 38 of the 50 (KramaBench 18, LakeQA 20) have a single answer under every reasonable reading. Most of the other 12 do not pin down a subset, such as which beaches count as "daily-monitored", or do not say which of two versions of the same dataset in the data lake to use. They still exercise the data lake's preparation and discovery work, and since TEAL-Q's answers are never used, they cannot act as wrong labels.

Thank you for the detailed review. Your questions led to several new analyses, including the gold audit of the verifier (Q1), the paired per-task test (Q3), the Explorer ablation on LakeQA (Q4), and the audit of TEAL-Q questions (Q6). We hope these responses address your concerns, and we would be grateful if you could take them into account in your assessment. We are happy to answer any further questions.

Reviewer_te9hOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TEAL, a framework that evolves reusable tools, persistent data artifacts, and usage instructions for coding agents performing data-lake analytics without ground-truth answer labels. So instead of redoing data discovery and cleaning for every question, the agent builds a reusable library of tools and usage guidance. An Explorer proposes tools from the lake itself, while a Critic proposes improvements from Worker trajectories. Proposed changes undergo implementation, skill updates, and rerun-based verification. TEAL-Q generates synthetic questions for this process. Experiments on KramaBench and LakeQA report improvements across four Workers under tight step budgets, transfer across agent harnesses, and comparable results from evolution on real versus synthetic questions. The paper also argues that evolving libraries under tighter budgets produces better tools for deployment under looser budgets.

Critical strengths
  1. Important, practical problem: Coding agents over data lakes redo the same discovery, cleaning and joining work for every question, and labeled answers are expensive to get in this domain. Evolving reusable tools without labels addresses a real and common inefficiency.

  2. Complementary tool-evolution strategy: Combining data-lake exploration with feedback from Worker trajectories offers complementary ways to identify useful tools. Synthetic-question training and cross-harness transfer further strengthen the approach’s practical usefulness.

Critical weaknesses
  1. Libraries evolved at high budgets sometimes fall below the baseline: At 20–30 steps, a library evolved at the same budget sometimes underperforms the plain Worker; Gemini-3.8-Flash on LakeQA drops from about 61 to about 49 at 30 steps. This raises the question of which budget tools should be evolved at for maximum benefit. The paper recommends evolving at a tight budget, but this might vary heavily across models and settings, and it doesn’t explain how to choose the budget. It also raises practical concerns about deployment. A practitioner has no labels to tell in advance which evolve budget will work, and tuning it would require exactly the labels the method claims to avoid. Deployment budgets also vary, so a poorly chosen budget could leave the agent worse off than with no library at all.

  2. Insufficient validation of verification and tool quality: No controlled ablation isolates the verifier’s contribution, and accepting inconclusive judgments or retaining changes when only one rerun passes may allow harmful tools. A manual audit would help establish whether accepted tools are correct and useful, beyond usage frequency. It could also explain whether the accuracy degradation from libraries evolved under looser step budgets stems from faulty tools, misleading usage instructions, or verification errors.

Questions for authors

Please see weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer te9h [1/2]

Q1: Choosing the budget in deployment

Thank you for this constructive question. In short, we recommend evolving at a tight budget at which the library saves steps and deploying it unchanged at any looser budget, which needs no labels.

Deployment budgets also vary, so a poorly chosen budget could leave the agent worse off than with no library at all.

If we understand correctly, the concern is that a library evolved at a loose deployment budget may fall below the baseline. However, the evolution budget does not need to match the deployment budget. The deployment budget is set by practical needs, since looser budgets are more accurate but more expensive (Fig. 3), and it may vary. TEAL evolves the library once at a tight budget and deploys it unchanged at whatever looser budget is needed. In Fig. 3, every such frozen library (orange dashed) stays above the baseline at every looser budget, for every Worker on both benchmarks. The practitioner thus only needs to choose a tight evolution budget, which we address next.

The paper recommends evolving at a tight budget, but this might vary heavily across models and settings, and it doesn't explain how to choose the budget... tuning it would require exactly the labels the method claims to avoid.

The evolution budget should be tight but not too tight, so that the Worker is pressed for steps but can still finish its pipelines (Sec. 4.2). A label-free heuristic is to pick a tight budget at which the library saves steps. Saving steps shows that the library has compressed the Worker's pipelines, and this compression carries over to looser budgets. The table gives the steps saved, (baseline - TEAL) / baseline, over three test repeats.

WorkerKB k=15LakeQA k=10k=15k=20k=25
Sonnet-525.7%-22.4%-5.7%12.4%11.0%
Opus-4.825.1%-15.3%-0.4%9.3%16.3%
Gemini-3.1-Pro15.0%-4.9%13.4%14.2%14.2%
Gemini-3.8-Flash2.1%1.9%3.0%3.5%7.8%

Every budget used in the paper (in bold) saves steps, and its library stays above the baseline at every looser budget (Fig. 3). Two other step-saving libraries also stay above the baseline at every looser budget. Sonnet-5 is currently unavailable to us, so the KramaBench example uses Gemini-3.1-Pro.

Gemini-3.1-Pro, KBk=10k=15k=20k=25k=30
Baseline36.6455.4860.1865.9169.98
Paper library (evolved at k=15)-63.7467.8370.9571.41
Library evolved at k=10 (6.2% saved)58.0162.2167.7473.7671.33
Opus-4.8, LakeQAk=20k=25k=30
Baseline57.0858.7561.67
Paper library (evolved at k=25)-67.9268.33
Library evolved at k=20 (9.3% saved)61.6762.5065.42

We acknowledge that this choice is not always optimal, but it stays above the baseline and needs only step counts on unlabeled questions. We will add this analysis to the paper.

...Gemini-3.8-Flash on LakeQA drops from about 61 to about 49 at 30 steps.

This Worker differs from the other three as follows.

  • Extra steps add little. Its 10-step baseline already reaches 87% of its 30-step accuracy (30% to 49% for the other Workers), with a third of the steps (8.9 vs. 27.0).
  • It often runs out of steps without writing an answer, despite the instruction to answer in the final steps (L318-319). In the LakeQA k=30 baseline, 13% of answers are empty, vs. at most 4% for every other Worker in every cell.
  • Its looser-budget library prescribes a longer workflow, e.g., on LakeQA, eight stages at k=30 vs. two tool calls at k=15 (search_catalog, then query_files). This coincides with more empty answers (28% vs. 13% for the k=30 baseline).
AuthorsOpenReview ↗

Response to Reviewer te9h [2/2]

Q2: Reliability of verification and tool quality

No controlled ablation isolates the verifier's contribution, and accepting inconclusive judgments or retaining changes when only one rerun passes may allow harmful tools.

We compare verification with two alternatives. Replacing it with a gold-answer oracle, with the rest of TEAL unchanged, gives similar accuracy (72.73 vs. 72.54 on KramaBench, Table 3). In contrast, ISE (Table 1) keeps every tool that executes, with the same Worker and the same Opus-4.8 evolution model. It is 5.7 points below TEAL on KramaBench and 10.0 points below the baseline on LakeQA, which suggests that execution-only acceptance is not enough.

Check 3, in which the Analyst compares the original answer without the library change with the rerun answer with it, is the only check that compares answers. To check whether verification admits harmful changes, we audited its 295 judgments in the 40 evolution runs post hoc against the gold answers of the training questions, which TEAL never sees.

  • When exactly one of the two answers is correct, the Analyst favors it in 86 of 101 cases (85%). A change whose rerun answer is wrong while the original was right passes check 3 in only 11 of 219 passes (5.0%).
  • In 70 of 80 inconclusive cases (88%), both answers are wrong, so accepting the change cannot lower accuracy on that question. Among the rest, accepting helped in 7 cases and hurt in 3.

TEAL keeps a change if at least one rerun passes all three checks, because the Worker is stochastic. Two reruns of the same question with the same library give different answers in 34% of pairs (345 of 1,010), so requiring every rerun to pass would reject changes because of Worker noise rather than the tool.

A manual audit would help establish whether accepted tools are correct and useful, beyond usage frequency...

We audited all 19 tools of the two final Sonnet-5 k=15 libraries. We re-ran each tool on the inputs it received at test time and checked its outputs against results we computed ourselves with pandas.

  • Correctness. Where a tool returns a result, its output matches ours, and in the episodes we checked, no wrong answer came from a wrong tool output. At test time, bugs made only 15 of 326 library calls fail on KramaBench and 1 of 2,836 on LakeQA, and the Worker recovered from each failure.
  • Usefulness beyond call frequency. On LakeQA, the tasks where the Worker calls rank_datasets_by_concept are hard ones (baseline 14.3%), and TEAL solves 44.3% of them (23 wins, 2 losses). On KramaBench, TEAL's gain is +11.7 points on tasks that use load_report_table vs. +7.5 on the rest.

As discussed in Q1, the drops of Gemini-3.8-Flash come mainly from empty answers under the longer workflow that a looser-budget library prescribes, not from faulty tools.


Thank you for your thoughtful feedback, which has helped us strengthen the paper. We hope our responses have addressed your concerns, and we are happy to answer any further questions.

Thanks for your detailed response to my concerns. I appreciate the manual auditing of the verifier and tools, it addresses my concern regarding verification and tool quality. For choosing the budget, clarification that library evolved at tighter budget can be deployed at looser budget is helpful. The proposed method of choosing tightest budget which gives step-saving seems reasonable but I am not sure about its reliability and the optimality concern also remains. Some discussion around that should be included in the manuscript and also how in future one can establish it.

In the light of the above response, I will raise my score to 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.