- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies how an agent can expand the range of numerical questions it can answer over structured records by generating new computational tools without reference answers. It proposes two loops that generate questions, identify missing capabilities, and add new operators to a domain specific language (DSL). New operators are checked through agreement between implementations and conservation relations. Experiments on merchant transactions, BookSQL, and EHRSQL report increased coverage, supported by mutation tests, ablations, and human evaluation.
Critical strengths
-
The paper addresses the important problem of expanding an agent's computational capabilities while checking newly generated operators.
-
The paper provides detailed prompts, sandbox rules, and validation procedures. These details help readers examine how the proposed checks are implemented and understand their scope.
Critical weaknesses
-
The presentation makes the task, contributions, and execution procedure difficult to follow. (a) The abstract introduces terms such as “disjoint starting sets” and “outside the starting closure” without explanation. (b) Section “Introduction” mixes the validation principle, framework, method components, and experimental results across six contributions. (c) Lengthy captions also repeat the surrounding discussion. (d) Section “3.2 OUTER LOOP” describes gap identification, operator authoring, validation, and library updating in that order. Figure 1 shows a different order, with arrows leading from validation to operator authoring and then to gap identification. These presentation problems make the main contribution and actual procedure difficult to assess.
-
In Section “Related Work”, the authors discuss LILO, TroVE, and LATM. However, the experiments do not directly compare the proposed framework with these methods. The reported comparisons mainly cover simplified admission rules and a SQL baseline. The advantage over existing methods for learning libraries and generating tools remains unclear.
Questions for authors
-
Which describes the actual procedure, Figure 1 or the text in Section “3.2 OUTER LOOP”?
-
What advantages does the proposed method offer over recent baselines such as LILO, TroVE, and LATM?
2 comments
Thanks to the authors for their response. Given the inconsistency between Figure 1 and the procedure described in the text, and the need for broader comparisons with related methods across different models, I will keep my score.
Response
Thank you for the clear and specific comments. Below, we address W1 (a) to (d) together with Q1, and then W2 together with Q2.
W1/Q1. Presentation and the actual procedure.
The text in Section 3.2, together with Algorithm 1, describes the actual procedure:
We will redraw Figure 1 to follow this order with numbered stages, which also addresses W1(d).
We will define both terms where they first appear. A "starting set" is a few question types together with exactly the operators that answer them, and its "closure" is everything those operators can compose.
Put simply, the paper makes one claim and supports it in three steps. The claim is that an agent can grow its own exact DSL with no answer key, reward, or judge, as long as it admits a new operator only on evidence it can check by itself (agreement with an independent implementation, plus conservation laws). The steps are the two-loop method that applies this rule, evidence that the rule is sound (none of 138 buggy variants is admitted), and evidence that the loop grows what the agent can answer across seeds, domains, and model families. In the camera-ready version of the paper, the contribution list will follow this structure, so each item makes one point.
We will shorten the captions to what is needed to read each figure or table.
W2/Q2. Comparison with LILO, TroVE, and LATM.
To answer this directly, we ran a comparison of complete systems. We adapted TroVE and LATM to our setting and ran each inside the same loop, with the same model, gap discovery, drafting prompt, and sample budget, so that only the admission rule differs. TroVE-style admission keeps the shortest draft whose answers to the gap questions agree with the majority of drafts. LATM-style admission keeps any draft that passes unit tests it wrote itself, since no labeled examples exist in this setting. We also measure each system's accuracy on held-out, human-written questions over 1,000 unseen merchants.
Buggy variants are single-point mutations of each system's own admitted operators. We will add this comparison to the camera-ready version of the paper.
Our method grows the largest answerable set, answers the most held-out questions correctly, and is the only one that admits no buggy operator. The difference comes from what each rule checks. A majority vote and self-written tests only see how an operator behaves on the inputs they happen to cover. Our gate instead compares two independent implementations on every validation record and on edge cases, and it also checks conservation laws stated over trusted operators. LILO keeps the abstractions that compress the library best, which corresponds to the reuse/compression rule in our admission study; that rule admits 101 of 138 buggy variants.
We hope the new comparison answers your question, and we are happy to discuss further.