acceptodds
Under review as a conference paper at ICLR 2027

RelCraft: Credit-Guided and Point-in-Time-Safe Program Search for Relational Learning

Abstract

Large Language Model (LLM) agents can automate relational feature engineering by searching over executable SQL, but free-form generation often produces invalid or temporally unsafe programs, scalar validation feedback poorly identifies which feature block helped, and repeated reuse of one validation set causes optimistic selection. We introduce RelCraft, a reliability-oriented framework with three matched components. A schema-typed grammar and deterministic point-in-time verifier restrict search to executable, row-aligned programs. Paired cross-fitted coalition evaluation estimates each block's average marginal utility under matched model fits. A protected split then selects a frozen SQL–predictor pair using a fixed complexity prior without returning selection outcomes to the LLM. We give a finite-sample guarantee for the bounded group loss used by this selector and evaluate predictive quality, search behavior, explicit future-edge violations, and deployment cost on 32 tasks from three relational benchmark suites.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Relational Data Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ggVsOpenReview ↗
Rating
2(+1)
Confidence
3(+2)
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper proposes RelCraft, a framework for automated feature engineering in relational learning. The method uses a typed, point-in-time-checked program representation to generate and verify SQL features. It introduces coalition-based credit assignment to guide LLM-driven program search and employs a protected selection mechanism with a statistical risk bound to select the final program and predictor. The authors evaluate RelCraft on several relational learning benchmarks, comparing predictive performance and efficiency against existing methods. They also conduct ablation studies to examine the contributions of the proposed components.

Critical strengths
  1. The paper addresses a practical problem in relational learning by integrating program verification, point-in-time constraints, and feature search into a unified framework.

  2. The experimental evaluation covers multiple relational learning benchmarks and includes ablation studies and efficiency analyses, providing useful evidence of the effectiveness of the proposed framework.

Critical weaknesses
  1. Presentation: The mathematical notation is not consistently presented, and several important definitions are scattered across the paper. For example, Eqs. (2), (8), and (10) use optimality notation for solutions obtained under different objectives.

  2. Theoretical Assumptions: The theoretical guarantee relies on the independence of entity-group losses. However, relational databases naturally contain dependencies through shared entities, relationships, and temporal effects. The paper does not establish whether this assumption holds in the evaluated settings, raising concerns about the applicability of the theoretical guarantee.

  3. Experimental Evaluation: The experiments do not sufficiently compare the proposed coalition-guided search with alternative program search strategies under the same constraints and computational budget. This makes it difficult to assess the necessity of the proposed search mechanism.

  4. Methodological Validation: The paper does not clearly establish the assumptions and conditions under which coalition-based credit assignment can effectively guide program search, particularly when features exhibit strong interactions. These conditions should be explicitly formulated and empirically verified to support the central methodological claim.

Questions for authors

Questions For Authors:

  1. Theorem 1 assumes independent entity-group losses. Given the dependencies naturally present in relational databases, how is this assumption justified in the evaluated datasets? Are the resulting risk bounds sufficiently tight to be meaningful in practice?

  2. Under what conditions does coalition-based credit assignment effectively guide program search? In particular, how does the method discover strongly interacting features that have little predictive value individually?

  3. How does coalition-guided search compare with alternative search strategies, such as greedy search or MCTS, under the same program constraints and computational budget? What evidence establishes that the observed improvements specifically arise from the proposed credit assignment mechanism?

7 comments

We sincerely thank the reviewer for the careful reading and constructive suggestions, and for recognizing that RelCraft addresses a practical problem by integrating program verification, point-in-time constraints, and feature search, with evaluation across multiple benchmarks, ablations, and efficiency analyses. Below, Q denotes each question and R our response.


Q1. AI-use statement.

R: The previous version is replaced by the completed statement on Page 11: generative AI was used only for grammar, spelling, and typographical checks and for reference verification, and the authors reviewed all edits and take full responsibility for the content. We apologize for the oversight and hope this resolves the flag.


Q2. Eqs. (2), (8), and (10) use optimality notation for different objectives.

R: The revision separates the three objects explicitly (§3, Notation, Page 3). Eq. (2) defines the population target (Q∘,ϕ∘)(\mathcal Q^\circ,\phi^\circ). Search visits measured states hr=(Qr,ϕr)h_r=(\mathcal Q_r,\phi_r) and freezes a shortlist H\mathcal H. The protected selector returns h⋆∈Hh^\star\in\mathcal H (Eq. 8), and the deployed artifact is ϕ⋆\phi^\star fitted on Ttr∪Tdev\mathcal T_{\rm tr}\cup\mathcal T_{\rm dev} (Eq. 10).


Q3. Is the independence assumption justified, and is the bound tight enough to be meaningful?

R: The bound is non-vacuous and contains the test loss on all 32 tasks, and the observed gap stays within it under dependence-aware grouping. Theorem 1 is stated for independent protected entity groups (§4.4, Page 6). The new bound audit (Appendix E.6, Table 19, Page 30) measures it per task:

(1) At the entity level, the complexity term has median 0.095 and the bound median 0.272; the bound contains the test bounded loss on every task, exceeding it by 0.122 on average; the complexity term is at most 0.15 on 27 of 32 tasks.

(2) To probe dependence, we merge targets that share relational neighbors into clusters (155–5,726 per task) and treat each cluster as one group. This raises the complexity term by 1.59× on average and moves the Sel/Test gap only from 0.56 to 0.587 points. On every task, the upper 95% cluster-bootstrap limit of the protected–test gap (at most 0.95 points) stays below the cluster-level bound.


Q4. How does coalition-guided search compare with greedy search or MCTS under the same constraints and budget?

R: Coalition credit has the highest AUROC of six search rules, the smallest programs, and the best-calibrated selection. In the new search-rule control (§5.4, Table 3b, Page 8; Appendix E.2, Table 15, Page 28), all rules share the typed grammar, frozen candidate pool, verifier, protected selector, seeds, and 211 jobs; only the search rule changes. Coalition credit versus each alternative:

AlternativeΔAUROC (Holm pp)Fewer blocksSmaller Sel/Test gap
Random+1.04 (<0.001)10.50.40
Round-score greedy+0.71 (<0.001)9.10.32
LOO greedy+0.41 (0.235)7.20.23
Beam (width 4)+0.29 (0.235)4.80.17
MCTS+0.24 (0.291)4.40.15

All block and gap differences have Holm p<0.001p<0.001. Because every other component is held fixed, these differences arise from the credit assignment itself.


Q5. When does coalition credit guide search, and how does it find strongly interacting features with little individual value?

R: Coalition credit helps when a block's value depends on partners in the program universe. It scores each block in sampled contexts of every size (Eq. 5, Page 4), so a block is evaluated both with and without its partner. Leave-one-out credit scores a block in one context; an XOR block whose partner is absent gets zero gain and is pruned before the pair forms. This is why typed LOO search stays at 0.500 AUROC on XOR, while RelCraft reaches 1.000 (Table 5a, Page 9).

The new interaction sweep (Table 5b, Page 9; Appendix E.5, Table 18, Page 30) adds 1% label flips and six distractor features:

(1) With additive labels both rules agree (AUROC 0.988 vs. 0.984), so coalition credit costs nothing without interaction.

(2) The advantage grows with interaction strength: for XOR at M=8M=8, the AUROC gain over LOO rises from 0.11 at strength 0.25 to 0.45 at full strength.

(3) Higher-order interactions need more contexts: for fourth-order parity, Sign-F1 exceeds LOO from M=8M=8 (0.787 vs. 0.722; 0.836 at M=16M=16).

With the default M=8M=8, coalition credit beats LOO on both AUROC and Sign-F1 in all 12 interacting settings.


We hope the completed disclosure, the clarified notation, and the new controls address the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score.

Reviewer_ggVsOpenReview ↗

Response to reviewer

Thank you for the detailed response and additional experiments. I appreciate the improvements, but I still have several concerns.

  1. Some notation inconsistencies remain. Equation (6) uses C(b)C(b) and log⁡(1+τ)\log(1+\tau), while Appendix D.2 describes normalized quantities. I also do not understand how hat τb\tau_b in Eq. (7) is estimated or why a prediction is needed when execution time can be directly measured. In addition, why does Figure 2(c) use LCBδLCB_\delta, while Eq. (7) uses LCBηctx\mathrm{LCB}^{\mathrm{ctx}}_\eta? Which notation is correct?

  2. I appreciate the new comparisons. However, the improvements over MCTS and beam search in predictive accuracy are not significant. While the smaller programs and reduced selection–test gap are positive, I am still not fully convinced of the advantage of coalition credit over these stronger search strategies.

  3. The new grouping analysis partially addresses my concern about independence. However, I am not sure whether the grouping is recursive, and even disjoint clusters do not necessarily imply independence. I understand that perfect independence is difficult to establish, but some discussion of residual dependence or its effect on the bound would be helpful. More importantly, the bound still seems extremely loose. For example, Table 19 allows a margin of 10.4 percentage points when the observed protected–test difference is only 0.2 points. I find it difficult to see the practical significance of such a loose bound.

Overall, the current evidence and responses have not changed my original concerns regarding the notation, theoretical, and experimental support for the proposed method. Therefore, I maintain my original rating.

We thank the reviewer for the follow-up and for confirming that coalition credit yields smaller programs and a smaller selection–test gap. Below we show that the scales are fixed in the paper, that coalition credit leads every alternative search rule on every outcome of Table 3b, and that the 10.4-point margin comes from the grammar-wide prior: on the frozen shortlist the same argument gives 2.0 points.


Q1. Notation in Eq. (6), Eq. (7), and Figure 2(c).

R: The scales are fixed in the submitted PDF; we collect them here.

(1) Eq. (6) writes the penalties in natural units. Appendix D.2 (Page 22) gives the scale used in code: C\mathcal C enters as C~=C/Cmax⁡\widetilde{\mathcal C}=\mathcal C/\mathcal C_{\max}, and log⁡(1+τ)\log(1+\tau) as rτ=log⁡(1+min⁡{τsec,900})/log⁡901r_\tau=\log(1+\min\{\tau_{\rm sec},900\})/\log 901. D.2 states that the main-text log⁡(1+τ)\log(1+\tau) is implemented by rτr_\tau; both terms lie in [0,1][0,1]. The unnormalized C\mathcal C is used only in the prior π0\pi_0.

(2) τ^b\widehat\tau_b in Eq. (7). The measured time of bb already enters LCB⁡ηctx(b)\operatorname{LCB}^{\rm ctx}_\eta(b) through Eq. (6). Eq. (7) chooses which path family to expand next. The blocks created by that expansion have not run yet, so D.2 and Table 9 (Page 22) specify a prediction on the rτr_\tau scale. τ^b\widehat\tau_b is the mean measured rτr_\tau of the nbn_b evaluated blocks in bb's path family.

(3) Figure 2(c) shows the same score in schematic form. Its caption (Page 3) states that LCBδ\mathrm{LCB}_\delta denotes LCB⁡ηctx\operatorname{LCB}^{\rm ctx}_\eta of §4.2, so t=η=1t=\eta=1 and SEb=sΔ/MSE_b=s_\Delta/\sqrt M.


Q2. Advantage over beam and MCTS.

R: With grammar, candidate pool, verifier, selector, and the 211-job budget held fixed, coalition credit is ahead of every alternative on every outcome of Table 3b. The differences in the deployed program are significant against all three.

vs.ΔAUROC [95% CI]Fewer blocksLatency (s/10K)Sel/Test gap
LOO greedy+0.41 [0.07, 0.75]7.2 (32/32)−8.1% (29/32)−0.23
Beam (4)+0.29 [0.05, 0.52]4.8 (31/32)−5.9% (26/32)−0.17
MCTS+0.24 [0.01, 0.48]4.4 (30/32)−5.9% (30/32)−0.15

(1) Accuracy. All three unadjusted 95% AUROC intervals exclude zero. With Holm correction over all search-rule contrasts, pp = 0.235–0.291, so §5.4 claims accuracy parity. AUTC$_N$ is significantly higher than for LOO and beam (Holm pp = 0.013).

(2) Deployed program. Block, latency, and gap differences all have p<0.001p<0.001. Latency uses the Table 15 runs. At the same budget and accuracy, MCTS and beam deploy 14–15% larger programs whose protected estimates track test less closely.

(3) Mechanism. When block value depends on partners, the accuracy gap is large. On XOR, coalition credit gains +0.45 AUROC over LOO, and it has higher Sign-F1 in all 12 interacting settings (Table 5, Page 9; Table 18, Page 30).


Q3. Grouping, residual dependence, and tightness.

R: (1) Grouping. Yes: clusters are connected components (the transitive closure) of the graph linking protected targets that share a row through a declared PK–FK edge. Point (2) covers what remains, such as time trends.

(2) Residual dependence. Suppose group losses have a dependency graph with fractional chromatic number χ∗\chi^*. Then Hoeffding's inequality holds with nsel/χ∗n_{\rm sel}/\chi^* in place of nseln_{\rm sel} (Janson, RSA 2004). ρδ\rho_\delta grows by a factor of χ∗\sqrt{\chi^*}, and the union argument is unchanged.

(3) Tightness. The 10.4-point margin reflects log⁡π0−1=123\log\pi_0^{-1}=123–171171 nats, the union over the whole grammar H0\mathcal H_0 that Eq. (8) uses to penalize long programs. Theorem 1 conditions on the frozen H\mathcal H, so a uniform union over ∣H∣=14|\mathcal H|=14–3030 is valid under the same assumptions, and every input is in Table 19 (Page 30):

ρδH=(log⁡∣H∣+log⁡δ−1)/(2nsel)\rho^{\mathcal H}_\delta=\sqrt{(\log|\mathcal H|+\log\delta^{-1})/(2n_{\rm sel})}

  • At the entity level, ρδH\rho^{\mathcal H}_\delta has median 1.9 points (range 1.6–9.1). On 4DB cla-01 it is 2.0 instead of 10.4. At the cluster level, the median is 2.9 points.
  • The observed ∣L^−Lte∣|\hat L-L_{\rm te}| is 0.33 points on average. Its cluster-bootstrap 95% upper limit (at most 0.95) lies below the cluster-level ρδH\rho^{\mathcal H}_\delta on all 32 tasks.
  • ρδHχ∗\rho^{\mathcal H}_\delta\sqrt{\chi^*} equals the Table 19 value at χ∗=20\chi^*=20–3030. A certificate no larger than the reported bound therefore holds even if each protected group depends on up to 19 others.

In practice the selector keeps the gap small: without it, Sel/Test rises from 0.56 to 1.41 points (Table 4, Page 9).


The final version would add the shortlist certificate and χ∗\chi^* result after Theorem 1 and the latency to Table 15. With the ethics concern resolved by the completed AI-use statement, we would be grateful if the reviewer would reconsider the score.

Reviewer_YVfFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RelCraft, an LLM-agent framework for relational feature engineering that targets three failure modes of existing SQL-writing agents: invalid or temporally unsafe programs, ambiguous scalar feedback, and adaptive overfitting to the validation set. RelCraft (i) has the LLM propose typed feature sketches that are deterministically compiled to SQL and verified for strict point-in-time cutoffs and row alignment, (ii) estimates each feature block’s contribution with Shapley-style, paired, cross-fitted coalition credit, and (iii) selects the final SQL–predictor pair once on a protected split hidden from the agent, with a complexity prior and a finite-sample bound on the bounded selection loss.

Critical strengths
  • The failure modes of existing work are clearly defined.
  • The deployment strategy is interesting, which is w/o the help of LLMs.
Critical weaknesses
  • Baselines are from published results. I recommend the authors to rerun the baseline results instead of directly applying the exising results from other paper
  • Compute is not matched. RelCraft uses 52% more cross-fit jobs than RelAgent (211.4 vs. 139.0).It is actually not a fair comparison.
  • Some evaluations are directly obtained from the algorithm design. For exmaple, the time leakage is obtained from the framework initial design.
Questions for authors

See the weakness.

5 comments

We sincerely thank the reviewer for the careful reading and constructive suggestions, and for recognizing the clearly defined failure modes of existing work and RelCraft's LLM-free deployment. Below, Q denotes each question and R our response. All new results use the common harness (shared snapshots, split rows, predictor menu, seeds, and DuckDB limits) with five seeds; effects are paired over tasks with 10,000 whole-task bootstrap resamples, 50,000 sign flips, and Holm correction within each test family.


Q1. Compute is not matched: RelCraft uses 211.4 vs. 139.0 cross-fit jobs.

R: At exactly equal compute, RelCraft remains significantly more accurate than RelAgent at every budget. In the new compute-matched reruns (§5.4, Table 3a, Page 8; Appendix E.1, Table 14, Page 28), both agents run with exactly 139, 211, or 240 unique cross-fit jobs per task–seed (each job trains five forward-chaining fold models). In the default runs the gain is +0.91 on 20 of 21 tasks (Holm p<0.001p<0.001).

Jobs per task–seedRelAgent AUROCRelCraft AUROCPaired gain [95% CI]Tasks improvedHolm pp
13980.8081.41+0.61 [0.34, 0.91]16/210.010
21181.0981.71+0.62 [0.25, 0.99]15/210.035
24081.1681.75+0.59 [0.30, 0.86]17/210.011
RelCraft 211 vs. RelAgent 24081.1681.71+0.55 [0.25, 0.88]16/210.020

(1) At every budget, RelCraft is better on all 32 of 32 tasks in SQL failure (−4.5 to −4.8 pp), Dev/Test gap (−0.44 to −0.74 points), and Sel/Test gap (−0.68 to −0.88 points), all with Holm p<0.001p<0.001. Search efficiency (AUTC$_N$) is also higher at every budget (Holm p≤0.011p\le0.011).

(2) Extra budget does not close the gap. From 139 to 240 jobs, RelAgent gains 0.36 points and RelCraft 0.34, and RelCraft at 139 jobs (81.41) already exceeds RelAgent at 240 jobs (81.16).

(3) The extra default jobs are the paired context evaluations of coalition credit: without coalition credit, RelCraft uses exactly 139.0 jobs (Table 4, Page 9).

(4) NMAE is lower at every budget, by 0.014–0.016 on 8–10 of 11 regression tasks. With only 11 tasks this is not significant after correction at matched budgets (Holm pp = 0.052–0.065); in the default runs it is (0.020).


Q2. Baselines are taken from published results; please rerun them.

R: Every paired claim in the revision uses baselines rerun under our common harness. Table 1 (Page 7) is kept only to place RelCraft against published suite aggregates and carries no significance claim. The reruns are:

(1) RelAgent, RelAgent+guard (RelAgent with a prompt guard), and ReFuGe on all 32 tasks (Table 2, Page 8):

Rerun methodSQL fail (%)Dev/TestSel/Test#BlocksLatency (s/10K rows)
ReFuGe9.51.671.4954.39.1
RelAgent6.61.541.3147.17.4
RelAgent+guard4.81.421.1844.57.1
RelCraft2.01.020.5631.75.8

(2) RelAgent at three matched budgets (Table 3a, Page 8; Q1 above).

(3) Five alternative search rules under a shared grammar, candidate pool, and selector (Table 3b, Page 8).

(4) RelAgent under three LLM backbones (Appendix Table 12, Page 24): RelCraft improves AUROC on every backbone, from 80.79 to 81.71 (GPT-5.2), 80.22 to 81.24 (DeepSeek-V4), and 79.33 to 81.00 (Qwen3-32B).


Q3. The leakage evaluation follows directly from the framework design.

R: We added an audit of the declared failure modes on natural outputs of a strong baseline. The synthetic Leak/2K probe is a conformance check of the verifier against its specification. The new audit (Appendix E.7, Table 20, Page 31) applies the verifier to the SQL that RelAgent actually selected on the 32 benchmark tasks:

(1) The verifier flags 206 of 7,536 RelAgent features (77 cutoff violations, 135 row inflations, 6 both), with at least one flagged feature on every task (0.44–4.51% per task).

(2) Manual review confirms 61 of 64 flagged features as unsafe (95% precision) and finds only 2 unsafe features among 128 random unflagged ones.

(3) RelCraft's 5,077 selected features have no flags, and all 192 sampled ones are safe.

(4) A prompt guard does not enforce these constraints: RelAgent+guard lowers SQL failure only from 6.6% to 4.8%, whereas typed compilation and static verification reach 2.0% (Table 2, Page 8).

These failures occur in programs a strong baseline actually deploys, and RelCraft removes them while also improving AUROC.


We believe these controlled reruns directly address the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score.

Thank you for the detailed response and additional experiments. The compute-matched comparisons substantially address my concern about unequal cross-fit budgets, and I appreciate the baseline reruns under a common evaluation harness. The audit of naturally generated SQL also provides more informative evidence than the original synthetic conformance check.

I acknowledge that the additional experiments strengthen the paper. However, the comparative evidence summarized in the response is not yet sufficient to change my overall assessment, so I am maintaining my original rating at this stage.

We thank the reviewer for the follow-up and for confirming that the compute-matched comparisons address the budget concern, that the baselines are rerun under a common harness, and that the natural-SQL audit is more informative than the synthetic check. Below we collect the comparative evidence in one place, ordered from the strongest baseline to the published field.


(1) Against RelAgent at equal compute, the gain exceeds what extra compute buys. At 139, 211, and 240 jobs, RelCraft improves AUROC by 0.59–0.62 points (Holm pp = 0.010–0.035; Table 3a, Page 8; Table 14, Page 28). RelAgent gains 0.36 points when its budget grows from 139 to 240 jobs, so RelCraft's matched-budget advantage is 1.6–1.7 times that gain. RelCraft at 139 jobs (81.41) is ahead of RelAgent at 240 jobs (81.16) while using 42% fewer jobs. NMAE is also lower at every budget, on 8–10 of 11 regression tasks.

(2) The gain holds on every LLM backbone and is largest on the weakest. RelCraft improves AUROC by 0.92 (GPT-5.2), 1.02 (DeepSeek-V4), and 1.67 points (Qwen3-32B) (Table 12, Page 24). RelCraft with the open 32B model (81.00) is ahead of RelAgent with GPT-5.2 (80.79). Its AUROC range across backbones is 0.71 points, against 1.46 for RelAgent.

(3) The deployed program is more reliable than that of every rerun agent. At each matched budget, relative to RelAgent, RelCraft lowers SQL failure by 4.5–4.8 pp, the Dev/Test gap by 0.44–0.74 points, and the Sel/Test gap by 0.68–0.88 points, on 32 of 32 tasks (Holm p<0.001p<0.001). In the default runs it also leads RelAgent+guard and ReFuGe on every search, reliability, and deployment column, with 31.7 blocks against 44.5–54.3 and 5.8 s/10K latency against 7.1–9.1 (Table 2, Page 8).

(4) Against alternative search rules, the gain comes from the feedback. With grammar, pool, verifier, selector, and 211 jobs fixed, coalition credit beats random and round-score greedy search by 1.04 and 0.71 AUROC (Holm p<0.001p<0.001), and deploys 4.4–7.2 fewer blocks with a 0.15–0.23 point smaller Sel/Test gap than LOO, beam, and MCTS (Holm p<0.001p<0.001; Table 3b, Page 8; Table 15, Page 28).

(5) Against the published field, RelCraft is best on four of five suite metrics (Table 1, Page 7). Relative to KumoRFM-2, the strongest non-agent method, it is higher by 2.23 AUROC on RB-V2 and 2.30 on 4DBInfer and lowers NMAE from 0.822 to 0.793 and from 0.601 to 0.528, while staying within 0.26 points on RB-V1.


These results cover the strongest agentic baseline at equal compute, three backbones, three rerun agents, five alternative search rules, and seven published baselines. If a specific comparison would still change the assessment, we would be glad to address it in the discussion; otherwise we would appreciate it if the reviewer would reconsider the score.

Reviewer_Zh26OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RelCraft, the LLM-agent framework for relational feature engineering with SQL, following the existing literature RelAgent. It targets three failure modes for RelAgent: free-form SQL could be invalid or temporally unsafe, a scalar validation score cannot identify which feature block helped, and repeated reuse of one validation set leads to optimistic selection. RelCraft proposes a typed sketch grammar with a point-in-time verifier, cross-fitted coalition credit for each feature block, and a protected split for the final selection, with a finite-sample guarantee for a bounded loss. RelCraft reports small gains over the published RelAgent results. In matched reruns, it lowers the SQL failure rate, the protected–test gap and the number of selected blocks, at the cost of 52% more cross-fit jobs.

Critical strengths

Each of the three failure modes is paired with a targeted solution, and I appreciate the careful identification of existing agentic solutions. The component is also evaluated with its own targeted measures, with disciplined evaluation and carefully scoped results.

Critical weaknesses

The improvements are mainly on the engineering side, and the theoretical result adds little. The three failure modes do not inherently connect with each other, and the only link between the components is a shared complexity term. Each component also builds on familiar tools, e.g., Deep Feature Synthesis-style enumeration along key paths, sampled Shapley credit, and a held-out split with a complexity prior. Additionally, some of the solutions are already implicitly implemented in RelAgent. For example, RelAgent provides a dedicated evaluation workspace, which already goes beyond the simple scalar feedback. The paper wraps them with mathematical illustrations, but Theorem 1 is a standard Hoeffding and union bound on a bounded surrogate loss rather than on AUROC or MAE. As a result, it is unclear what the main conceptual contribution is beyond the engineering integration.

Additionally, empirically, the improvement is modest, and the ablation results seem mild. Compared with the published RelAgent results, RelCraft improves AUROC by less than one point on each classification suite, and it remains slightly below KumoRFM-v2 on RelBench V1. The ablation losses are also modest: removing the static verifier, the protected split or the complexity prior only changes AUROC by less than 0.3 points, which is close to the seed variation of the full model. I would appreciate additional experimentation to show that the introduced methodology is indeed significant enough to support the claim.

Questions for authors
  1. How were α in Eq. 4 and the other fixed coefficients in Table 9 chosen? The values seem a bit arbitrary, and the paper only states that they were fixed before benchmark evaluation. Since they directly affect which programs are selected, it would be helpful to know whether they were tuned on any of the evaluated tasks.

  2. The schema-typed grammar may limit the agent's expressiveness by restricting it to a smaller search space (e.g., no subqueries or free-form literals). Have the authors considered this an issue? Specifically, what fraction of the SQL features selected by RelAgent can be expressed in the grammar?

  3. The proposed modifications target failure modes observed on the evaluated benchmarks, which have tables with declared keys and timestamps. Can they generalize to other relational datasets or settings?

  4. The AI use statement is currently a placeholder. Could the authors clarify their use of AI tools?

6 comments

We sincerely thank the reviewer for the careful reading and constructive suggestions, and for recognizing that each failure mode is paired with a targeted solution and the disciplined, carefully scoped evaluation. Below, Q denotes each question and R our response. All new results use the common harness with five seeds, task-level paired tests (whole-task bootstrap, sign flips), and Holm correction within each test family.


Q1. What is the main conceptual contribution beyond engineering integration?

R: The contribution is a search-feedback protocol for relational feature programs, and the new search-rule control isolates its effect. Each block is scored by paired base–augmented comparisons across sampled program contexts, with folds, preprocessing, predictor, and seed fixed within each pair (§4.2, Eq. 5, Page 4). Typed compilation defines the admissible edits (§4.1, Page 3), and a one-way protected boundary keeps final selection out of the feedback loop (§4.4, Page 5). RelAgent provides an evaluation workspace and LLM-free deployment, as §2.2 (Page 2) states; RelCraft changes what is measured for each edit and how that measurement drives search.

In the new search-rule control (§5.4, Table 3b, Page 8), grammar, candidate pool, verifier, protected selector, and 211 jobs are shared and only the search rule changes. Coalition credit improves AUROC by 1.04 points over random search and 0.71 over greedy search, and gives the smallest programs and the smallest Sel/Test gap of all six rules. Theorem 1 is a standard Hoeffding–union argument, as the paper states; its role is to calibrate the protected choice, and it is non-vacuous on all 32 tasks (Appendix E.6, Table 19, Page 30).


Q2. The improvement over RelAgent and the ablation effects are modest.

R: Under paired task-level inference the gains are significant, hold at equal compute, and are large on the reliability metrics.

(1) At equal budgets of 139, 211, and 240 jobs, the AUROC gain is 0.59–0.62 points with 95% intervals that exclude zero (Table 3a, Page 8; Table 14, Page 28). In the default runs it is 0.91 points, on 20 of 21 tasks (Holm p<0.001p<0.001).

(2) On all 32 tasks, RelCraft lowers the Sel/Test gap from 1.31 to 0.56 points, SQL failure from 6.6% to 2.0%, and program size from 47.1 to 31.7 blocks (Table 2, Page 8).

(3) Each mechanism is significant on its pre-declared primary outcome (Table 4, Page 9; Table 16, Page 29): without the typed grammar, SQL failure rises from 2.0% to 8.8%; without coalition credit, the Sel/Test gap grows by 0.35 points and the program by 10.5 blocks; without protected selection, the Sel/Test gap rises from 0.56 to 1.41 points; without the complexity prior or execution cost, the program grows by 18.7 or 15.6 blocks at unchanged AUROC.


Q3. How were α\boldsymbol\alpha and the coefficients in Table 9 chosen?

R: One tuple of round values, (λc,λτ,λe,η,κ,γ)=(10−3,10−3,10−3,1,10−2,1)(\lambda_c,\lambda_\tau,\lambda_e,\eta,\kappa,\gamma)=(10^{-3},10^{-3},10^{-3},1,10^{-2},1), is fixed across all tasks, suites, and backbones before benchmark evaluation (Table 9, Page 22). The new sensitivity study (Appendix E.4, Table 17, Page 29) halves and doubles each coefficient and α\boldsymbol\alpha. Across these 14 perturbations, AUROC stays within 81.58–81.73 (default 81.71), NMAE within 0.7447–0.7462, and Sel/Test within 0.55–0.63 points. The coefficients mainly set program size (26.9–34.1 blocks).


Q4. What fraction of RelAgent's selected SQL can the grammar express?

R: 94.5% (7,122 of 7,536 features; 91.4–96.9% per task) (Appendix E.7, Table 20, Page 31). The remainder are window/order statistics (2.1%), non-local correlated subqueries (1.6%), recursive paths (0.9%), UDFs (0.5%), and set operations (0.4%). Since RelCraft is more accurate than RelAgent at every matched budget, the typed language gives up little predictive signal. Window and order statistics, the largest category, are a direct typed extension listed as future work (§7, Page 10).


Q5. Can the method generalize to other relational datasets?

R: RelCraft requires exactly the metadata that any point-in-time feature pipeline needs: PK–FK keys, event timestamps, and entity identifiers with cutoffs. Every RelBench and 4DBInfer task already declares them. The evaluation covers 32 classification and regression tasks across three suites and three LLM backbones (Appendix Table 12, Page 24).


Q6. The AI-use statement is a placeholder.

R: The completed statement is on Page 11: generative AI was used only for grammar, spelling, and typographical checks and for reference verification; the authors reviewed all edits and take full responsibility for the content.


We hope the new controls make the contribution and its measured effect clear, and we would be grateful if the reviewer would consider raising the score.

I thank the authors for the detailed rebuttal, additional experiments, and clarification of the paper's main conceptual contribution.

However, I still have concerns regarding the paper's positioning and methodological novelty. The authors now emphasize the search-feedback protocol as the central contribution, which is an interesting research direction in itself. If this is the intended focus, I encourage the authors to motivate it more explicitly in the introduction, rather than framing the paper around three relatively independent failure modes.

Nevertheless, the proposed approach largely integrates established techniques, including Shapley-style coalition credit, cross-fitted evaluation, and protected holdout selection. While Table 3b shows improvements over random and round-score greedy search, the AUROC gains over stronger alternatives (leave-one-out greedy, beam search, and MCTS) are not statistically significant after Holm correction (Table 15). Similarly, none of the component ablations shows a statistically significant AUROC improvement on RelBench V1 after correction (Table 16). Thus, I remain unconvinced that the proposed search-feedback protocol represents a substantial methodological advance.

Alternatively, if the main contribution is positioned around reliability and efficiency, I acknowledge the practical improvements, but view them primarily as engineering refinements of the existing RelAgent framework rather than new methodological insights.

I appreciate the strengthened evaluation, but my concern regarding the paper's conceptual novelty remains. I therefore maintain my original rating.

We thank the reviewer for the follow-up, for finding the search-feedback protocol an interesting direction, and for acknowledging the practical gains. Below we show that the paper already centers this protocol, that the other two components are its validity conditions, and that it gives search a capability the alternatives lack.


Q1. Positioning: the protocol versus three failure modes.

R: The introduction already names the protocol as the central contribution, and the other two components are its preconditions. Page 2 states: "Our central methodological focus is the search feedback." The protocol scores one edit by paired base–augmented comparisons, and that score is valid only under two conditions:

(1) Admissibility. A paired difference is meaningful only if both programs execute and stay row-aligned with the targets; otherwise a leaking or row-inflating block earns credit. Typed compilation guarantees both before any credit is computed: free-form SQL admits 187 of 2,000 post-cutoff sketches, typed generation none (Table 5a, Page 9).

(2) One-way selection. Finer feedback fits the development split more closely, so the final choice must sit outside the loop. Without it, the Sel/Test gap rises from 0.56 to 1.41 points (Table 4, Page 9).

The final version will open the introduction with the feedback problem and present admissibility and protected selection as its two preconditions, as the reviewer suggests.


Q2. The ingredients are established.

R: The new element is what search measures for each edit, and it changes which programs search can reach. Shapley weights usually explain a fixed model after training. Here they define the feedback an LLM receives while editing a program, computed by paired cross-fits with folds, preprocessing, predictor, and seed matched within each pair (Eqs. 5–6, Page 4). Existing agents revise from one whole-trial score (Figure 1b, Page 2).

The consequence is qualitative. Under forward search, a block that is useful only with a partner gets zero gain from leave-one-out feedback and is pruned before the pair forms. Coalition credit scores it in contexts that contain the partner. Only the credit rule differs:

ProbeTyped + LOOCoalition
XOR AUROC (Table 5a)0.5001.000
XOR, noisy sweep, M=8M=8 (Table 5b)0.5000.948
Parity-4, M=8M=8 (Table 5b)0.5000.874
Sign-F1 (Table 5a)0.7310.914

Whole-trial feedback also gives 0.500 on XOR. With additive labels both rules agree (0.984 vs. 0.988; Table 18, Page 30), so the gain appears exactly where the mechanism predicts.


Q3. AUROC significance in Tables 15 and 16.

R: Against prior methods the AUROC gain is significant. LOO, beam, and MCTS are RelCraft with only the search rule replaced.

(1) Prior methods. At 139, 211, and 240 jobs, RelCraft beats RelAgent by 0.59–0.62 AUROC (Holm pp = 0.010–0.035; Table 14, Page 28), and wins on every LLM backbone (Table 12, Page 24). Against greedy search on the whole-round score, the Figure 1b signal, the gain is +0.71 (Holm p<0.001p<0.001).

(2) Strengthened variants. LOO greedy, beam, and MCTS already use RelCraft's typed grammar, verifier, frozen pool, protected selector, and 211 jobs. Coalition credit is still ahead of all three on all four Table 3b outcomes, and its programs are 12–19% smaller with a 21–29% smaller Sel/Test gap (Holm p<0.001p<0.001). Equal accuracy with fewer blocks means the credit identifies which blocks carry the value, which is what the protocol measures.

(3) Ablations. Table 4 assigns each component a primary outcome, and each is significant there (Table 16, Page 29). Removing coalition credit adds 10.5 blocks, 0.35 Sel/Test points, 4.2 pp SQL failure, and 0.86 s/10K latency, on 31–32 of 32 tasks (all Holm p<0.001p<0.001).


Q4. Reliability and efficiency as refinements of RelAgent.

R: RelCraft keeps RelAgent's deployment form, SQL plus a tabular predictor, and changes the three decisions that existing systems make "inside one loop of trials on shared development data" (Page 1). Each change gives a property that tuning the loop cannot:

  • Typed compilation guarantees point-in-time safety for every admitted sketch. RelCraft's 5,077 selected features have no flags, while 206 of RelAgent's 7,536 are flagged and 61 of 64 sampled flags are confirmed unsafe (Table 20, Page 31).
  • Protected selection carries a finite-sample bound (Theorem 1, Page 6).
  • Coalition credit recovers interactions that leave-one-out and whole-trial feedback miss (Q2).

RelAgent's engineering fix, a prompt guard, lowers SQL failure only from 6.6% to 4.8%, against 2.0% for RelCraft (Table 2, Page 8).


We hope this clarifies that the contribution is a change in what search measures, with testable consequences, and we would be grateful if the reviewer would reconsider the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Esteemed Area Chair and Reviewers,

We sincerely thank all reviewers for their thoughtful questions, constructive suggestions, and positive comments on RelCraft. We appreciate their recognition of:

  • A practical problem with clearly defined failure modes, each paired with a targeted solution (All reviewers).
  • A meaningful contribution: coalition-based credit for LLM-guided relational program search, with an LLM-free deployment (Reviewers Zh26 and YVfF).
  • Disciplined evaluation with convincing improvements in reliability and efficiency (All reviewers).

Following the discussion, Reviewers ggVs and Zh26 raised their ratings. Reviewer YVfF acknowledged that the additional experiments strengthen the paper.

Novelty. Existing relational agents revise SQL from one whole-trial score; RelCraft gives the agent per-edit credit for each block it writes:

  • Coalition credit over a growing universe: blocks are scored across coalitions that include same-round proposals, so partners are credited before either joins the program (Cohen et al. (2005) select from a fixed set).
  • Valid by construction: matched paired cross-fits, typed admissibility, and protected selection.

In response to their questions and suggestions, we provided the following key clarifications and additions:

QuestionReviewerOur Response
How does RelCraft perform in fair, matched comparisons?YVfF, Zh26RelAgent (including its Table 1 row), RelAgent+guard, and ReFuGe are rerun in our harness on all 32 tasks. At exactly 139, 211, and 240 cross-fit jobs, RelCraft stays 0.59–0.62 AUROC ahead of RelAgent (Holm p ≤ 0.035), and RelCraft at 139 jobs exceeds RelAgent at 240 (Table 14, Page 29). It also leads on every LLM backbone (+0.92, +1.02, +1.67 AUROC; Table 12, Page 25). Reviewer YVfF confirmed that this substantially addresses the budget concern and appreciated the reruns.
Does the verifier catch leakage in real agent-written SQL?YVfFThe verifier is applied to the SQL RelAgent actually selected: 206 of 7,536 features flagged, 61 of 64 sampled flags confirmed unsafe, against 0 of 5,077 for RelCraft (Table 20, Page 33). The reviewer found this audit more informative.
What does each component contribute?Zh26Against RelAgent, SQL failure falls from 6.6% to 2.0% and the Sel/Test gap from 1.31 to 0.56 (Table 2, Page 8). Each component is significant on the outcome it targets: without typed grammar SQL failure rises to 8.8%, without protected selection the Sel/Test gap rises to 1.41, and without coalition credit the program grows by 10.5 blocks (Table 4, Page 9; Table 16, Page 31). The reviewer found the improvements in reliability and efficiency convincing.
Does coalition credit beat other search rules under the same budget?ggVs, Zh26With grammar, pool, verifier, selector, and 211 jobs fixed, coalition credit gains +1.04 and +0.71 AUROC over random and round-score search, and deploys 4.4–7.2 fewer blocks, 5.9–8.1% lower latency, and a 0.15–0.23 smaller Sel/Test gap than LOO, beam, and MCTS (all Holm p < 0.001; Table 15, Page 30).
How does it find features that are useful only together?ggVsContexts include same-round proposals, so both XOR blocks receive positive credit before either joins the program (Appendix B.2, Page 17). XOR 0.500 → 1.000, Parity-4 0.500 → 0.874 against typed LOO (Tables 5a–5b, Page 9); the lead grows from +0.11 to +0.45 with interaction strength (Table 18, Page 32).
How does the guarantee handle relational dependence, and how tight is it?ggVsThe bound contains the test loss on all 32 tasks; PK–FK cluster grouping and a fractional-chromatic extension (Janson, 2004) cover dependence, and the shortlist certificate has median 1.9 points (Page 6; Appendix E.6, Page 28).
Is the presentation clear and consistent?ggVsNotation is unified throughout, with a full notation table (Appendix H, Page 35), and the writing is revised for clarity. Reviewer ggVs acknowledged the improvements and the additional evidence and raised the rating.

We thank the reviewers and the AC again for helping strengthen the paper. We hope these clarifications and additional results support favorable consideration of RelCraft as a principled search-feedback protocol for LLM agents that write relational feature programs.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.