acceptodds
Under review as a conference paper at ICLR 2027

A Self-Improving Agent that Grows its Own DSL: Oracle-Free Operator Induction for Exact Question Answering

Abstract

A self-improving agent has to decide which new skills to keep, and in practice it keeps whatever earns a high reward, convinces a judge model, or compresses its library, even when the skill is wrong. That is fine until the answers have to be exactly right: over a table of transactions, a total is either right or wrong. We ask a harder question: can an agent grow what it can compute with no reward, no judge, and no answer key, and still be exactly right? A single frozen model runs two nested loops over a small, exact domain-specific language (DSL). The inner loop answers new questions only when two independently written expressions agree on every record. The outer loop discovers its own missing capability and authors an operator for it, admitted only when an independent second implementation agrees and the result obeys a conservation law, all without ground truth. The inner loop roughly triples the answerable-question set, and the outer loop self-extends on disjoint starting sets, inventing operators outside the starting closure; a no-induction control answers none of the new questions. The admission rule is empirically sound: it rejects 138 of 138 buggy operator variants under an independent reference, even one written by a different model family, where the admit-if-it-runs, self-consistency, and compression signals the field relies on admit most of them. Unmodified, the same mechanism self-extends on two further domains with unrelated schemas, business-ledger and clinical-encounter question answering, growing their answerable sets more than tenfold and nearly twentyfold. The loop transfers across the proposer's model family too: across further model families the gate stays sound, and independent families self-extend comparably, with only the induction yield tracking each proposer's DSL fluency.

Primary area:
foundation or frontier models, including LLMs
Keywords:
self-improving agents, LLM agents, domain-specific languages, program synthesis, operator induction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AzfFOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how an agent can expand the range of numerical questions it can answer over structured records by generating new computational tools without reference answers. It proposes two loops that generate questions, identify missing capabilities, and add new operators to a domain specific language (DSL). New operators are checked through agreement between implementations and conservation relations. Experiments on merchant transactions, BookSQL, and EHRSQL report increased coverage, supported by mutation tests, ablations, and human evaluation.

Critical strengths
  1. The paper addresses the important problem of expanding an agent's computational capabilities while checking newly generated operators.

  2. The paper provides detailed prompts, sandbox rules, and validation procedures. These details help readers examine how the proposed checks are implemented and understand their scope.

Critical weaknesses
  1. The presentation makes the task, contributions, and execution procedure difficult to follow. (a) The abstract introduces terms such as “disjoint starting sets” and “outside the starting closure” without explanation. (b) Section “Introduction” mixes the validation principle, framework, method components, and experimental results across six contributions. (c) Lengthy captions also repeat the surrounding discussion. (d) Section “3.2 OUTER LOOP” describes gap identification, operator authoring, validation, and library updating in that order. Figure 1 shows a different order, with arrows leading from validation to operator authoring and then to gap identification. These presentation problems make the main contribution and actual procedure difficult to assess.

  2. In Section “Related Work”, the authors discuss LILO, TroVE, and LATM. However, the experiments do not directly compare the proposed framework with these methods. The reported comparisons mainly cover simplified admission rules and a SQL baseline. The advantage over existing methods for learning libraries and generating tools remains unclear.

Questions for authors
  1. Which describes the actual procedure, Figure 1 or the text in Section “3.2 OUTER LOOP”?

  2. What advantages does the proposed method offer over recent baselines such as LILO, TroVE, and LATM?

2 comments

Thank you for the clear and specific comments. Below, we address W1 (a) to (d) together with Q1, and then W2 together with Q2.

W1/Q1. Presentation and the actual procedure.

Q1: "Which describes the actual procedure, Figure 1 or the text in Section '3.2 OUTER LOOP'?"

The text in Section 3.2, together with Algorithm 1, describes the actual procedure:

  1. identify the most requested missing capability;
  2. author a candidate operator;
  3. validate it with an independent second implementation and conservation relations;
  4. add it to the library;
  5. rerun the inner loop.

We will redraw Figure 1 to follow this order with numbered stages, which also addresses W1(d).

W1(a): "The abstract introduces terms such as 'disjoint starting sets' and 'outside the starting closure' without explanation."

We will define both terms where they first appear. A "starting set" is a few question types together with exactly the operators that answer them, and its "closure" is everything those operators can compose.

W1(b): "Section 'Introduction' mixes the validation principle, framework, method components, and experimental results across six contributions."

Put simply, the paper makes one claim and supports it in three steps. The claim is that an agent can grow its own exact DSL with no answer key, reward, or judge, as long as it admits a new operator only on evidence it can check by itself (agreement with an independent implementation, plus conservation laws). The steps are the two-loop method that applies this rule, evidence that the rule is sound (none of 138 buggy variants is admitted), and evidence that the loop grows what the agent can answer across seeds, domains, and model families. In the camera-ready version of the paper, the contribution list will follow this structure, so each item makes one point.

W1(c): "Lengthy captions also repeat the surrounding discussion."

We will shorten the captions to what is needed to read each figure or table.

W2/Q2. Comparison with LILO, TroVE, and LATM.

Q2: "What advantages does the proposed method offer over recent baselines such as LILO, TroVE, and LATM?"

To answer this directly, we ran a comparison of complete systems. We adapted TroVE and LATM to our setting and ran each inside the same loop, with the same model, gap discovery, drafting prompt, and sample budget, so that only the admission rule differs. TroVE-style admission keeps the shortest draft whose answers to the gap questions agree with the majority of drafts. LATM-style admission keeps any draft that passes unit tests it wrote itself, since no labeled examples exist in this setting. We also measure each system's accuracy on held-out, human-written questions over 1,000 unseen merchants.

Amounts/timing starting setOursTroVE-styleLATM-style
Answerable types after two rounds336273232
Accuracy on held-out questions (%)25.218.722.1
Buggy operator variants admitted0/1382/292/132

Buggy variants are single-point mutations of each system's own admitted operators. We will add this comparison to the camera-ready version of the paper.

Our method grows the largest answerable set, answers the most held-out questions correctly, and is the only one that admits no buggy operator. The difference comes from what each rule checks. A majority vote and self-written tests only see how an operator behaves on the inputs they happen to cover. Our gate instead compares two independent implementations on every validation record and on edge cases, and it also checks conservation laws stated over trusted operators. LILO keeps the abstractions that compress the library best, which corresponds to the reuse/compression rule in our admission study; that rule admits 101 of 138 buggy variants.

We hope the new comparison answers your question, and we are happy to discuss further.

Thanks to the authors for their response. Given the inconsistency between Figure 1 and the procedure described in the text, and the need for broader comparisons with related methods across different models, I will keep my score.

Reviewer_oqkXOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The manuscript targets agents that self-improve by evolving a library of tools when no external signals are available. It is assumed that the underlying model is frozen and that the tools are based on a limited domain-specific language (DSL). Based on a small, initial set of solvable tasks/questions and tools, these tasks and tools are expanded through two loops. The inner loop focuses on expanding the solvable tasks by proposing new tasks first and then trying to solve them through the existing tools. If tasks cannot be solved, the respective issue is recorded. Based on the most requested tools, the outer loop proposes new capabilities, which are validated through a rigorous process. First a second implementation for the tool is written and both of them are empirically validated, i.e., the initial proposal only proceeds to the next step, if their outputs match on a number of inputs. Afterwards conversation laws are applied and checked. Finally, the inner loop is rerun and it is checked, whether the new capabilities help to solve more tasks. Their approach is validated on three different datasets with a number of model sizes and model families.

Critical strengths

The (self-)improvement of agents is a relevant and timely topic.

The evaluation is reasonable using three different datasets, different model sizes as well as families. It also includes some ablation studies.

Critical weaknesses

The writing is the biggest issue with this manuscript. This manifests in two direction: the language itself as well as the structure, i.e. how the research is presented. The writing is very rough language-wise which makes it hard to understand what is written. It also not very precise and rigorous, for example contains phrases like "compress beautifully". Initially I thought the manuscript might have been initially written in a non-English language, and subsequently been machine-translated to English, however according to the AI statement that is not true.

On a more structurally level, the manuscript does not tell a good story instead information is presented indifferently to the reader. This starts with the fact that the setting, that manuscript is trying to solve is unclear. My guess is some form of data analytics over relational databases. Abstract and introduction read like summaries of the manuscript at different levels. Of course they should provide such information, but not only. For example, the introduction should also motivate their work within the current research landscape and explain the relevant setting for context. The description is also very fuzzy and scarce of technical details. As an example, the following questions arose for myself just from Section 3.1, which is less than half a page:

    1. Propose: "draw many samples: -> How is the final question selected?
    1. Execute:
    • What is an "outside" operator? Do you mean here, that the model hallucinates operators?
    • Where does the validation set come from?
    1. Verify:
    • "model then answers the question text alone, separately" -> Figure 1 and earlier discussion seem to indicate that another expression is used, while this implies that models answers directly.
    • What does it mean separately?
    1. Keep if new: Here the term "quantity" is used, but I think actually (question, expression) pair is meant.
  • no discussion why this architecture was chosen
  • Why are these question, expression pairs important?
  • How does it "name the missing capability" work?

Some of these questions are answered in the appendices, which are also being very unstructured (there are 25 different high-level sections in the appendix), so again the reader has to search for the relevant information. While the main document contains links to the respective appendices, there are so many of them, that it is unclear to the reader, when a detour is necessary and when not. To summarizes, it feels like most relevant information is actually somewhere in the manuscript, but needs to be presented much better.

Similarly, the evaluation section suffers from the aforementioned lack of clear general setting. But also the experimental setting is partially unclear, since most results are discussed in the main body, however the experiments themselves and how they are setup are "hidden" in the appendices. The reader should have at least a notion about the respective experiments before their results are analyzed. The writing in the evaluation chapter can also be improved in general. For example, 4.5 claims to introduce experiments on two new datasets, but they have already been discussed in 4.4 as well as earlier figures. While the evaluation might be reasonable general, these failures makes the whole chapter hard to understand. Additionally a discussion of the hyperparameters and how they were derived is missing. Another issue is that there is no "external" validation. While different datasets are used, much of their evaluation stays within the logic how the framework is applied to the evolving agents and measures the same signals. The "final" agents are never taken and applied to external benchmarks/tasks.

I did not understand how these "conservation laws" work, but that might be a reflection on myself.

minor:

  • Figure 1:
    • the inner arc of the outer loop is aligned with arrow from Expand Answerable Set
    • order of component execution for the outer loop is unclear from the figure alone
  • line 754: "This is how we discover what is missing." might not be part of the prompt
Questions for authors

major questions:

  • What is the purpose of the framework since there is not outside interaction, i.e. what is it trying to solve?
  • Where do the conversation laws come from and how are they applied?
  • Have you applied the a final checkpoint of the agents to some external benchmarks?

minor questions:

  • Where do the functions come from?
  • What is the knowledge base?
  • How are the initial questions and operators determined?
  • Any notion on why Gemini does not add any induced operators?
5 comments
AuthorsOpenReview ↗

Response Part 1

Thank you for the careful and detailed reading. To make our answers easy to match to your comments, we number your weaknesses W1 to W10 and your questions Q1 to Q7 in the order they appear in your review. A few questions overlap with weaknesses (Q1 with W2, Q2 with W8, and Q3 with W7). We answer each of these in full once and point to that answer from the other place.

W1. The language is rough and imprecise.

"The writing is very rough language-wise ... It also not very precise and rigorous, for example contains phrases like 'compress beautifully'."

We are sorry the writing made the paper hard to read. In the camera-ready version of the paper, we will remove informal phrases and go through the text sentence by sentence to make it precise and plain.

W2. The paper does not tell a clear story, and the setting is unclear.

"...the manuscript does not tell a good story ... the setting, that manuscript is trying to solve is unclear. My guess is some form of data analytics over relational databases."

Your guess is close. The setting is exact numerical question answering over one entity's structured records, such as a merchant's transaction history, a business ledger, or a patient's billed clinical events. A frozen LLM answers by writing a single expression in a small, sandboxed DSL, and a deterministic interpreter evaluates it, so a correct expression always gives an exactly correct number. Coverage is the limit: the DSL can only answer what its operators can compose.

Here is the story the paper tells. Self-improving agents usually keep a new skill because it earned a reward, convinced a judge model, or made their library more compact. None of these signals can tell a correct skill from a subtly wrong one, which matters when answers must be exact. We ask whether an agent can grow its own exact DSL with no answer key, reward, or judge. Our answer is to admit a new operator only on evidence the agent can check by itself: an independently written second implementation must agree with it on every record, and it must obey conservation laws stated over operators that are already trusted. A frozen model applies this rule in two loops, one that finds new answerable questions and one that adds operators. The experiments then show that the rule is sound (all 138 buggy variants are rejected) and that the loops grow what the agent can answer across starting sets, domains, and model families, including on held-out questions written by people (W7).

In the camera-ready version of the paper, the opening paragraphs will follow this story, and the introduction will place the work relative to program-of-thought methods and to library learning (LILO, TroVE, LATM).

AuthorsOpenReview ↗

Response Part 2

W3. The method description lacks technical detail (your Section 3.1 questions).

"The description is also very fuzzy and scarce of technical details. As an example, the following questions arose for myself just from Section 3.1..."

Thank you for listing these questions. They show exactly where the text needs more detail, and we will add the answers below to Section 3.1.

  • (a) Propose: "How is the final question selected?" No single question is selected. Every sampled (question, expression) pair goes on to execution once duplicate expressions are removed.
  • (b) Execute: "What is an 'outside' operator? Do you mean here, that the model hallucinates operators?" Yes. An outside operator is one that is not in the current library but that the model reaches for anyway. Calling it raises an error, so the pair is dropped.
  • (c) Execute: "Where does the validation set come from?" It is a fixed set of 63 synthetic merchants, shared by every phase of a run.
  • (d) Verify: "...another expression is used ... What does it mean separately?" Your reading is right: a second expression is used. In a fresh prompt, the same model reads only the question and writes a second expression without seeing the first, and this is what "separately" refers to. The pair is kept only if both expressions agree on every validation merchant.
  • (e) Keep if new: "...the term 'quantity' is used, but I think actually (question, expression) pair is meant." Correct, we keep (question, expression) pairs. By "quantity" we meant the vector of the expression's values over the 63 merchants. A pair counts as new when this vector differs from all known ones, so rewordings of the same computation are counted only once. We will use "pair" and "value vector" consistently.
  • (f) "...no discussion why this architecture was chosen." We will add a short motivation for each stage. Round-trip agreement stands in for an answer key, and the novelty check keeps rewordings from inflating coverage.
  • (g) "Why are these question, expression pairs important?" They form the knowledge base, which records what the agent can already answer and supplies examples for later steps.
  • (h) "How does it 'name the missing capability' work?" While proposing questions, the model may call an operator that does not exist yet. We count these names across samples, and the most requested one becomes the gap.

W4. The appendix is hard to navigate.

"...the appendices, which are also being very unstructured (there are 25 different high-level sections in the appendix)..."

We will add a short guide at the start of the appendix that groups its sections by topic: method details, experimental setup, additional results, and prompts. The main text will point to these groups, so readers can tell when a detour is needed.

W5. Experimental setups are hidden in the appendix, and Sections 4.4 and 4.5 overlap.

"...the experiments themselves and how they are setup are 'hidden' in the appendices..." / "...4.5 claims to introduce experiments on two new datasets, but they have already been discussed in 4.4..."

We agree. In the camera-ready version of the paper, each results subsection will open with one or two sentences on how its experiment is set up, and we will remove the repetition between Sections 4.4 and 4.5 so that each dataset is introduced once.

W6. Hyperparameters and how they were chosen.

"Additionally a discussion of the hyperparameters and how they were derived is missing."

The appendix gives the default settings: 63 validation records, 64 samples per generation, 4 independent answers per question, temperature 1.0, at most 8 inner passes per phase, the top 3 gaps per induction round, and one repair attempt per draft. Section 4.1 reports that the inner loop keeps finding new types across 8 different sampling-budget settings, and the gate's outcome stays the same from 3 to 63 validation records (0/138 buggy variants admitted). In the camera-ready version of the paper, we will move these settings into a table in the main text and explain how each was chosen.

AuthorsOpenReview ↗

Response Part 3

W7. The final agents are not validated externally.

"Another issue is that there is no 'external' validation ... The 'final' agents are never taken and applied to external benchmarks/tasks."

We ran this evaluation. We froze the agent after each stage and gave it held-out, human-written questions that the loop never generated: 40 merchant question types, each in three paraphrases (120 questions), on 1,000 synthetic merchants that the loop never saw. The agent answers each question with a single DSL expression, using its current library and 8 retrieved verified examples. An answer counts as correct when it matches the value of the type's canonical expression.

Agent (amounts/timing starting set)Accuracy (%)
Seed library17.5
No induction, same inner-loop budget19.7
TroVE-style admission18.7
LATM-style admission22.1
Ours, after two rounds25.2

The induced operators let the agent answer question types that the seed library cannot. For example, accuracy on the top buyer's purchase count rises from 0% to 100%. A matched-budget control also shows that the growth in answerable types comes from induction itself:

Answerable types after two roundsOursNo inductionNo induction, +1 phase
Amounts/timing seed336289318
Timing seed299266292
EHRSQL248205221

We will add both tables to the camera-ready version of the paper.

W8. How the conservation laws work.

"I did not understand how these 'conservation laws' work, but that might be a reflection on myself."

The model writes them alongside each candidate operator: one to four boolean relations stated only in terms of the new operator and operators that are already trusted. For a per-group total, for instance, the relation sum(per_group_total(buyer_ids, amounts)) == sum(amounts) requires the per-group totals to add back up to the overall total. The harness keeps a relation only if it calls both the new operator and a trusted operator and contains a comparison. It then checks the relation on every validation record, and a single failure rejects the operator. These relations add protection beyond the second implementation: even when the same bug is planted in both implementations, the relations alone still reject 128 of 138 buggy variants.

W9 (minor). Figure 1.

"the inner arc of the outer loop is aligned with arrow from Expand Answerable Set" / "order of component execution for the outer loop is unclear from the figure alone"

We will redraw Figure 1 with numbered stages that follow Section 3.2 and Algorithm 1: identify the gap, author the operator, validate it, add it to the library, and rerun the inner loop.

W10 (minor). Line 754.

"line 754: 'This is how we discover what is missing.' might not be part of the prompt"

This sentence is part of the prompt. The listing reproduces the gap-discovery prompt verbatim, and the sentence tells the model why it should name a missing operator instead of forcing the existing ones to fit. In the camera-ready version of the paper, we will note that all prompt listings are verbatim.

Reviewer_pizLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a self-improving agent that autonomously expands its DSL for question answering without relying on ground-truth answers, rewards, or external judges. The framework introduced a loop that discovers new answerable questions by composing existing operators, and another loop that invents new operators from missing capabilities. The authors proposed a verification mechanism that combines agreement between independently generated implementations. Evaluating on synthetic merchant transactions, the framework demonstrates large growth in answerable question types.

Critical strengths
  • I found the use of differential testing and metamorphic relations is very principled and bringing these to oracle-free self-improvement is very creative and also the result demonstrates empirical effectiveness. The use of these has the potential to benefit many more domains/tasks that rely on similar setup.
  • The authors conduct pretty extensive experiments across domains, model families, and starting operator sets, as well as ablation and mutation studies to examine the reliability of their proposed verification mechanism.
Critical weaknesses
  • the construction of answerable question types is too arbitrary and synthetic, which raises concerns about whether the reported coverage gains reflect transferable improvements in real-world qa capabilities.
  • the proposed verification mechanism seems to argue for computational consistency instead of semantic correctness, where authors' own evaluation reveals substantial semantic mismatches; this appears to weaken the central claim of reliable self-improvement that is oracle-free.
  • the claimed autonomous capability discovery appears heavily scaffolded by the predefined operator gaps and domain-specific induction prompts.
  • the paper is poorly-written and I found it extremely hard to comprehend and follow; authors admitted the use of AI in drafting part of the manuscript, but the presentation is very under-polished.
Questions for authors
  • How well do the reported improvements in answerable question types translate to performance gains on independently constructed, real-world QA benchmarks, rather than self-generated questions?
  • Given that only 84% of question expression pairs involving induced operators are judged semantically faithful, how can the proposed verification mechanism ensure that newly acquired capabilities are semantically correct?
  • To what extent does autonomous operator induction depend on the predefined capability gaps and domain-specific prompting?
4 comments
AuthorsOpenReview ↗

Response to W1/Q1 and W2/Q2

Response to Reviewer pizL (1/2)

Thank you for the careful review, and for finding our use of differential and metamorphic testing principled and our experiments extensive. Your three questions line up with your first three weaknesses, so we answer them in pairs (W1/Q1, W2/Q2, W3/Q3) and then turn to W4.

W1/Q1. Do the coverage gains carry over to independently written questions?

"How well do the reported improvements ... translate to performance gains on independently constructed ... QA benchmarks, rather than self-generated questions?"

Thank you for raising this. We ran a new experiment to answer it. We froze the agent after each stage and gave it held-out, human-written questions that the loop never generated: 40 merchant question types outside the starting set, each in three paraphrases (120 questions), asked on 1,000 synthetic merchants that the loop never saw. The agent answers each question with a single DSL expression, using its current library and 8 retrieved verified examples. An answer counts as correct when it matches the value of the type's canonical expression.

Agent (amounts/timing starting set)Accuracy (%)
Seed library17.5
No induction, same inner-loop budget19.7
Ours, after two rounds25.2

The improvement comes from the induced operators, which account for 7.1 of the 25.2 points. They let the agent answer question types that no combination of the seed operators can express. We will add this evaluation to the camera-ready version of the paper.

Held-out question typeSeedNo inductionOurs
Purchases by the top buyer00100
Share of transactions from the top buyer0069
Share of volume from the top-spending buyer16067

W2/Q2. Computational consistency versus semantic correctness.

"Given that only 84% of question expression pairs involving induced operators are judged semantically faithful, how can the proposed verification mechanism ensure that newly acquired capabilities are semantically correct?"

The gate certifies the operator itself: a new operator is admitted only if an independently written second implementation agrees with it on every record and its conservation laws hold. Blind human annotators agreed, unanimously, that this holds for all 53 induced operators.

The 84% figure measures something different: whether an automatically generated question is worded to match the expression written for it. It is about how a question was paired with its expression, not about whether the operators compute correctly. The external evaluation above tests end-to-end correctness, and there the induced operators answer independently written questions correctly (100% on the top buyer's purchase count). In the camera-ready version of the paper, we will report these properties separately, each with its own evidence:

PropertyEvidence
Operator correctness53/53 operators judged equivalent to an independent implementation; 0/138 buggy variants admitted
End-to-end answers25.2% vs. 19.7% on held-out, human-written questions; top-buyer purchase count from 0% to 100%
Question-expression faithfulness89% of sampled pairs judged faithful by blind annotators
AuthorsOpenReview ↗

Response to W3/Q3

W3/Q3. How much does induction depend on predefined gaps and prompting?

"To what extent does autonomous operator induction depend on the predefined capability gaps and domain-specific prompting?"

The gaps are not predefined. The model proposes questions freely, and whenever it calls an operator that does not exist, we record the name. The most requested name becomes the next gap.

To see how much the prompt matters, we removed every capability hint from the induction prompt: the group-by target list, the per-group example, the "return a list" instruction, and the conservation example. Even without these hints, the loop discovered its own gaps and self-extended on all three starting sets:

Starting setOperators admitted (no hints)Capabilities discovered
Amounts/timing13conditional counts, percentages
Top buyer9conditional counts, filtered means
Timing10half-period growth, top-k volume

We also started the loop from a library that already contains grouping operators. In that case, discovery moved on to new capabilities such as distinct counts and percentile slices, and the answerable set grew to 417 types, more than the 336 of the standard run. So self-extension does not depend on the capability hints. We will add this ablation to the camera-ready version of the paper.

AuthorsOpenReview ↗

Presentation

"...the paper is poorly-written and I found it extremely hard to comprehend and follow..."

We are sorry the paper was hard to read. Here is its main storyline in plain terms.

Agents that improve themselves usually decide what to keep based on a reward, a judge model, or how well a change compresses their library. That works when an answer only has to be good. Some questions, though, have exactly one right answer: ask how much a merchant's top buyer spent, and the total is either right or wrong. Our agent answers such questions by writing a short expression in a small calculator language (a DSL), so every answer is computed exactly. What it can answer is limited by its operators, so it has to add new ones over time. The hard part is adding them with no answer key, reward, or judge, and without ever letting a wrong one in.

One frozen model runs two loops to do this. The inner loop invents new questions that the current operators can already answer, and it keeps a question only when two independently written expressions agree on every record. The outer loop notices which operator the model keeps asking for, writes it, and admits it only if an independent second implementation agrees with it and it obeys simple conservation laws, such as per-group totals adding up to the overall total.

The experiments show three things. The loops grow what the agent can answer, including on held-out, human-written questions (W1/Q1). In our mutation tests, the admission rule rejects all 138 buggy operator variants. And the same loop works on two other domains and with other model families.

In the camera-ready version of the paper, we will define each term where it first appears, correct Figure 1, and shorten the captions.

We hope the new results answer your questions. Please let us know if anything is still unclear.

Thank you to the authors for the detailed rebuttal and additional experiments. I appreciate the effort to address my concerns.

For response to my W1/Q1, you mentioned some quantities such as 40 question types with 3 paraphrases per type and 1000 synthetic merchants. It is very hard to evaluate real-world implication as the criteria behind "human-written" is not discussed at all, and my request as in Q1 is to evaluate on some established real-world benchmarks instead of some ad-hoc human-written questions.

For response to my W2/Q2, a perfect operator correctness evaluation i.e., eq to some independent implementation and no buggy variants does not guarantee semantic correctness. Only 89% on judged faithfulness further supports the claim. I remain unconvinced on relevant claims written in the manuscript.

For response to my W3/Q3, I am curious how it compares when the original capability hints and domain-specific prompting are included in terms of "Operators admitted (no hints)" and "Capabilities discovered".

I remain substantially concerned on the clarity and presentation issue of the manuscript. Even in this rebuttal, I've found sentences and phrases that are too verbose yet vague to properly understand. I strongly encourage authors to directly respond using their own words, if any response has been ever refined by an AI tool.

Given the outlook of the current manuscript and the rebuttal, I maintain my rating and lean reject.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.