acceptodds
Under review as a conference paper at ICLR 2027

StepRel: Efficient Relational Feature Search beyond Full Interaction History

Abstract

LLM agents can construct relational features by exploring databases, generating SQL programs, and revising them through validation feedback. As search progresses, they accumulate useful knowledge alongside a growing history of candidate programs and intermediate results. Retaining this history repeatedly processes information that could instead be reused selectively. We introduce StepRel, which maintains database knowledge, the current SQL program, and search evidence outside the conversation. It updates programs through local edits and periodically rebuilds the context from the retained knowledge, program, and selected evidence. This separation preserves an exact program for further revision while carrying forward useful outcomes from earlier attempts. Across 18 RelBench tasks and four LLM backbones, StepRel maintains competitive predictive performance under a shared evaluation protocol while reducing the context needed for continued search. These results demonstrate that effective relational feature search can continue without carrying its complete interaction history.

Primary area:
foundation or frontier models, including LLMs
Keywords:
relational feature engineering, relational databases, LLM agents, context management

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_vpbLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes StepRel, a method for reducing context use in LLM-based relational feature search. It stores database knowledge, the current SQL program, and past search results outside the conversation. The agent updates individual SQL blocks and periodically rebuilds its context using the stored information. The goal is to maintain predictive performance without retaining the full interaction history.

Critical strengths
  • The paper addresses the growing context cost of LLM agents that repeatedly generate and evaluate SQL programs. Separating database knowledge, the current program, and search evidence is a practical way to preserve useful information while reducing repeated context processing.
  • The paper evaluates 18 relational prediction tasks across four LLM backbones. It compares predictive performance, stability, and token usage against existing methods. The experiments show that StepRel can maintain competitive predictive performance while substantially reducing context length.
Critical weaknesses
  • The method mainly combines existing ideas such as external state storage, local program updates, and context reconstruction. Appendix E evaluates database knowledge reuse and local updates, but does not isolate evidence selection or context reconstruction. This leaves the added value of the proposed state-management design unclear.

  • Although StepRel reduces context length, cumulative token savings vary across backbones and evaluation conditions. Appendix D shows that the benefits depend on brief-construction costs, search depth, and unusually long tool interactions. This limits the generality of the efficiency claims.

Questions for authors
  • Does the proposed evidence-selection strategy provide measurable benefits over a simpler baseline that uses the same database brief, current SQL program, and reconstruction schedule, but includes only recent evaluation results in the reconstructed context?

  • What is the effect of disabling periodic context reconstruction while keeping database knowledge, local program updates, and evidence selection unchanged?

  • At what search depths does StepRel achieve cumulative token savings across different backbones when the full database-brief construction cost is included for a single rollout?

3 comments
AuthorsOpenReview ↗

(1/2) Evidence selection and context reconstruction

Thank you for recognizing the practical value of separating database knowledge, executable programs, and search evidence, and for your questions about state management and token savings.

Q1–Q2/W1: Isolating evidence selection and history clearing.

StepRel maintains database knowledge (K), the incumbent SQL program (P), and cumulative search evidence (E). We expanded the ablations to address your questions directly:

  • For Q1, compare Full StepRel with Recent evaluation records (E). This control uses the same database brief, current program, local updates, and reconstruction schedule. It replaces cumulative structured evidence with recent evaluation records, including scores, feature changes, importance, and existing diagnostics, without surrounding dialogue.
  • For Q2, compare Full StepRel with E + full history. This control retains all prior interactions while preserving K/P/E maintenance and local updates. It disables history clearing to isolate the effect of context reconstruction.
  • Additional controls address the broader concern in W1. No precomputed brief (K) allows online inspection without upfront knowledge preparation. Full SQL output (P) replaces local edits with complete candidate SQL while retaining the single-incumbent, single-block restriction. Recent interaction history (E) replaces cumulative evidence with recent model/tool exchanges. The latter keeps the same K/P and reconstruction schedule as StepRel.

We evaluate all six methods on six tasks across four backbones (three classification and three regression tasks across four databases, selected for coverage and short runtimes).

Cells show mean test rank / total tokens (StepRel=100%), both lower being better. Ranks average task ranks after seed averaging. Costs sum task-mean brief and search input/output tokens. Overall weights backbones equally.

MethodDeepSeekGPTQwen3.6CoderOverall
Full StepRel3.67 / 100.0%2.67 / 100.0%1.83 / 100.0%3.33 / 100.0%2.88 / 100.0%
Recent evaluation records (E)3.00 / 93.3%2.17 / 87.9%3.67 / 92.2%2.67 / 168.1%2.88 / 110.4%
E + full history4.17 / 55.3%2.50 / 175.0%4.67 / 113.6%3.83 / 205.4%3.79 / 137.3%
No precomputed brief (K)3.17 / 112.0%4.00 / 73.2%2.83 / 124.8%4.00 / 218.2%3.50 / 132.1%
Full SQL output (P)3.50 / 193.1%4.83 / 230.5%5.00 / 195.2%3.50 / 249.7%4.21 / 217.1%
Recent interaction history (E)3.50 / 84.6%4.83 / 146.1%3.00 / 100.3%3.67 / 144.6%3.75 / 118.9%

Q1: Benefit over recent evaluation results. StepRel matches this baseline's overall predictive rank while using fewer tokens overall. Benefits vary across backbones, with the predictive advantage on Qwen and the aggregate cost advantage driven by Coder. The additional recent-interaction control has a weaker overall performance–cost balance.

Q2: Effect of retaining the full history. Clearing old interactions improves overall predictive rank and reduces overall token cost. It saves tokens on GPT, Qwen, and Coder, while retained history is cheaper on DeepSeek.

W1: Contribution of the complete design. The additional controls show that the precomputed brief improves overall rank and efficiency, although its upfront cost is harder to recover in some settings. Local edits substantially reduce tokens on every backbone and improve overall rank compared with full SQL output. Across all six methods, full StepRel achieves the joint-best overall predictive rank with the lowest overall normalized token cost. These results clarify the practical contribution of the components while retaining the backbone-dependent tradeoffs.

AuthorsOpenReview ↗

(2/2) End-to-end token costs

Q3/W2: At what search depth do savings cover the full database-brief cost?

We analyzed cumulative StepRel and RelAgent costs along their search trajectories, charging the full brief-construction cost to each StepRel rollout. This directly tests when search savings recover initialization and how the answer changes across backbones.

Costs use RelAgent=100%. Search change and brief overhead are measured relative to the same RelAgent total. The last column reports the first trial from which StepRel's mean cumulative cost stays below RelAgent through the observed endpoint. Coder brief costs are reconstructed offline.

Backbone / task typeSearch changeBrief overheadStepRel totalSavings sustained from trial
DeepSeek / classification-37.5%+12.5%75.0%14
DeepSeek / regression-38.1%+11.8%73.8%14
GPT-Luna / classification-31.7%+4.6%72.9%19
GPT-Luna / regression-11.0%+6.1%95.1%24
Qwen3.6* / classification-20.8%+10.8%90.0%17
Qwen3.6* / regression-17.0%+9.3%92.4%17
Qwen3-Coder* / classification-2.6%+12.0%109.4%Not observed
Qwen3-Coder* / regression-41.0%+5.2%64.1%3

Total tokens include cumulative input and output from search plus the full brief. Prompt length measures one call's input. Each curve uses the same matched run pairs throughout. A trial here means a completed candidate evaluation with a valid score.

Q3: When does StepRel become cheaper? Sustained savings begin at 14 trials for DeepSeek, 19–24 for GPT-Luna, and 17 for Qwen3.6. Coder regression recovers the brief cost by trial 3. Coder classification remains more expensive at the endpoint because its small search saving cannot cover brief generation. These are aggregate crossing points, not guarantees for every task.

(*) Both agents exhibited extreme loops with Qwen3.6, including 2,273 repetitions of a schema read by StepRel and 504 repetitions of a query by RelAgent. Coder also repeatedly requested tools after budget exhaustion despite explicit refusals, consuming tokens without new database evidence. Tool errors or refusals allow repair or abandonment without terminating the run, so these loops can continue. For starred results, we remove anomalous runs and their matched counterparts from the main-search comparison.

We thank the reviewer again for recognizing the practical motivation and breadth of the evaluation. Your questions prompted us to isolate the contribution of state management and examine when its efficiency benefits arise. We hope these additional experiments clarify both the value and the limits of the design and provide useful guidance for relational feature-search systems.

Thank you again for your constructive questions. We added controlled comparisons of evidence selection and history clearing, together with a fuller cost analysis including brief construction. We would appreciate any thoughts on whether these additions address your concerns and would be happy to clarify any remaining points.

Reviewer_ihDROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a novel LLM-based relational feature search, dubbed StepRel, that uses local program edits and periodically reconstructs the agents' context from its persistent states. The main motivation is to reduce repeated database exploration and regeneration of unchanged SQL to avoid accumulating conversation history and context. Compared with the previous approach, RelAgent, StepRel demonstrates reduced context requirements and competitive performance on 18 RelBench tasks.

Critical strengths

The paper addresses a reasonable practical limitation of agentic feature search, as to me, explicitly preserving executable programs and useful search evidence, while enforcing incremental updates, is a sensible way to avoid regenerating unchanged SQL and relying on an increasingly long conversation history. I also appreciate the appendix’s transparent reporting of initialization costs and sensitivity to unusually expensive runs.

Critical weaknesses

There are several concerns. First, the contribution is relatively incremental: StepRel mainly improves the search procedure of RelAgent through hierarchical decomposition, local updates, and context management. While these changes are useful, the underlying approach remains largely the same, and the methodological novelty seems somewhat limited.

Additionally, the total-cost savins are not as consisntent across the model. Although deepseek seems to show clear toekn savings in figure 4, in the appendix the authors have also demonstrated that, for example, for Qwen3-Coder, the database-brief overhead can outweigh the search savings, while for Qwen3.6, an expensive tool-loop run reverses the mean comparison. It would help to clarify when StepRel actually provides end-to-end savings.

I am a bit confused on why StepRel improves predictive performance over RelAgent. The component experiment suggests that local updates help, but it excludes the database brief and only covers four classification tasks with one backbone. I would appreciate additional explanation of the predictive improvements, as they are not fully explained.

Finally, Section 5.5 only provides two SQL examples. A more systematic comparison of feature counts, SQL-block counts, and query complexity would help understand whether StepRel produces similar programs more efficiently or changes their complexity and interpretability.

Questions for authors

How representative are the DeepSeek efficiency results of the other backbones? In Appendix D, could you clarify the conditions under which StepRel is expected to reduce total token consumption, rather than only prompt length? In particular, what explains the unusually high brief-construction cost for Qwen3-Coder?

What is your interpretation of the predictive gains over RelAgent? A discussion grounded in the existing results would be helpful.

Do the available runs reveal systematic differences between final StepRel and RelAgent programs in feature-column counts, SQL-block counts, or complexity?

4 comments
AuthorsOpenReview ↗

(1/2) Efficiency and component ablations

Thank you for recognizing the practical motivation and persistent-state design, and for your questions on efficiency and predictive gains.

Q1/W2: Total-token savings across backbones and Coder's overhead.

We compare four backbones to assess whether search savings cover brief generation, alongside the paper's predictive ranks. Costs use RelAgent=100% (Coder brief costs reconstructed offline).

BackboneTask typeSearch changeBrief overheadStepRel totalOverall rank: RelAgent → StepRel ↓
DeepSeekClassification-37.5%+12.5%75.0%3.062 → 2.625
DeepSeekRegression-38.1%+11.8%73.8%3.062 → 2.625
GPT-LunaClassification-31.7%+4.6%72.9%3.438 → 2.875
GPT-LunaRegression-11.0%+6.1%95.1%3.438 → 2.875
Qwen3.6*Classification-20.8%+10.8%90.0%5.750 → 4.562
Qwen3.6*Regression-17.0%+9.3%92.4%5.750 → 4.562
Qwen3-Coder*Classification-2.6%+12.0%109.4%8.938 → 6.500
Qwen3-Coder*Regression-41.0%+5.2%64.1%8.938 → 6.500

Total tokens include all input/output in the selected rollout and its brief. Prompt length measures one call's input.

With the loop handling below, total tokens fall in most settings, and predictive rank improves on all four backbones. Savings vary: Coder classification's small search saving cannot cover brief generation, leaving total cost higher.

(*) Both agents had extreme loops with Qwen3.6: StepRel repeated a schema read 2,273 times. RelAgent repeated a query 504 times. At 24 evaluations, total costs were 101× and 38× the means of other seeds for each task/method.

Tool errors/refusals allow repair or abandonment without terminating the run. Coder instead repeatedly requested tools after budget exhaustion despite explicit refusals, consuming tokens without new database evidence.

For starred results, we remove anomalous runs and their matched counterparts from the main-search comparison.

Q2/W3: Which components contribute to predictive gains and token savings?

StepRel maintains database knowledge (K), the incumbent SQL program (P), and search evidence (E). We test five controls on six tasks across four backbones (three classification and three regression tasks across four databases, selected for coverage and short runtimes):

  • Knowledge (K): remove the precomputed brief, allowing online inspection.
  • Program updates (P): output full SQL instead of local edits, retaining the single-incumbent, single-block restriction.
  • Search memory (E): StepRel maintains cumulative evidence across trials, organizing feature outcomes and diagnostics beyond a recent-history window. We replace this with recent interaction history (recent model/tool exchanges, including proposals, calls, and responses) or recent evaluation records (recent trial results and diagnostics, omitting surrounding interactions). E + full history keeps the same evidence maintenance alongside all prior interactions. All three retain K/P, local edits, immediate feedback, and checkpoint timing.

Cells show mean test rank / total tokens (StepRel=100%), both lower being better. Ranks average task ranks after seed averaging. Costs sum task-mean brief and search tokens. Overall weights backbones equally.

MethodDeepSeekGPTQwen3.6CoderOverall
Full StepRel3.67 / 100.0%2.67 / 100.0%1.83 / 100.0%3.33 / 100.0%2.88 / 100.0%
No precomputed brief (K)3.17 / 112.0%4.00 / 73.2%2.83 / 124.8%4.00 / 218.2%3.50 / 132.1%
Full SQL output (P)3.50 / 193.1%4.83 / 230.5%5.00 / 195.2%3.50 / 249.7%4.21 / 217.1%
Recent evaluation records (E)3.00 / 93.3%2.17 / 87.9%3.67 / 92.2%2.67 / 168.1%2.88 / 110.4%
Recent interaction history (E)3.50 / 84.6%4.83 / 146.1%3.00 / 100.3%3.67 / 144.6%3.75 / 118.9%
E + full history4.17 / 55.3%2.50 / 175.0%4.67 / 113.6%3.83 / 205.4%3.79 / 137.3%

Full StepRel achieves the joint-best overall predictive rank with the lowest overall normalized token cost.

K — Database knowledge. The brief improves overall performance and efficiency, although its upfront cost is harder to recover in some settings.

P — Local program updates. Local edits substantially reduce tokens across all four backbones compared with regenerating full SQL, while improving overall predictive performance.

E — Search memory. Organized evidence with history clearing gives a better overall performance–cost balance than recent interactions or E + full history. Recent evaluation records are a strong alternative: they match StepRel's overall rank but use more tokens overall. This supports the value of cumulative evidence maintenance, with benefits varying across backbones. Q3 examines how retained evidence can help revisit useful signals and develop more elaborate features.

AuthorsOpenReview ↗

(2/2) SQL analysis and predictive gains

Q3/W4: Does StepRel change the final SQL programs, and why?

We expanded the SQL analysis with aggregate statistics and additional case studies, examining both the final programs and the search trajectories that produced them. This also helps explain the predictive gains raised in Q2.

The table shows task-averaged results as RelAgent → StepRel. Feature counts exclude join keys.

BackboneFeature columnsSQL blocksRatio columns
DeepSeek42.12 → 59.5610.39 → 8.052.82 → 7.13
Qwen3.628.04 → 62.5417.87 → 9.641.04 → 4.96

StepRel produces more features within fewer SQL blocks, with more derived ratios. SQL inspection shows multiple time windows, conditional aggregates, distinct counts, and ratios sharing a block. The increased complexity lies in richer computations within blocks. The counts alone do not establish deeper joins or greater interpretability.

Our explanation is that StepRel helps the agent retain useful feedback and build on it. Evidence preserves signals from earlier attempts, including rejected candidates. The exact incumbent and local edits then allow the agent to refine the corresponding feature block while preserving the rest of the program. Extending a useful block with complementary counts, ratios, or time windows increases feature columns without requiring additional blocks.

This pattern is also visible during search: at the same evaluation depth, StepRel attempts more ratio-related revisions and accepts more of them as improved incumbents on both backbones. The final-program differences therefore accompany more exploration and successful refinement of these features.

A user-churn case shows how retained feedback leads to better features:

  1. Identify a useful signal. An activity-regularity block becomes the strongest block. The agent next tests shopping-day-of-week diversity. Although this feature receives 21.3% of reported importance, validation does not improve and the candidate is rejected.
  2. Carry the signal forward. After context reset, the agent explicitly cites the retained block diagnostics and proposes distinguishing sustained activity from concentrated purchasing. The candidate was rejected, but its diagnostic evidence remains available.
  3. Refine and validate. Within the same block, the agent adds distinct transaction dates and their ratio to transaction count, improving validation AUC from 0.7058 to 0.7084. Adding distinct active weeks and months then raises AUC to 0.7086. Both accepted revisions preserve the other seven blocks, and the distinct-date feature remains the highest-importance feature in the final program.

Importance guides exploration, while validation determines what to keep. This case illustrates how evidence from an unsuccessful attempt can inform a more effective feature formulation. We believe this is one contributor to the predictive gains, alongside database knowledge and other changes in search behavior.

We thank the reviewer again for recognizing the practical motivation and the value of preserving executable programs and search evidence while updating SQL incrementally. Your questions prompted us to examine both the benefits and the limitations more closely. The expanded experiments show that these design choices improve overall predictive performance and token efficiency, while the trajectory and SQL analyses help explain how they shape feature exploration. We hope these ablations and analyses will strengthen the understanding of StepRel and offer useful guidance for feature search and SQL analysis in other relational prediction tasks. We will incorporate these findings into the revision.

Thank you for the detailed rebuttal and additional experiments. My main remaining concern is the relatively incremental methodological novelty, as the approach largely combines established ideas in persistent state management, local program editing, and context reconstruction. Nevertheless, I believe the additional experiments substantially strengthen the empirical contribution and address most of my original concerns. I am therefore raising my rating from 2 to 3.

Thank you for carefully considering the additional experiments and for your updated assessment! We appreciate your feedback on the methodological contribution and will clarify its scope and incorporating the new results and analyses into the revision.

Reviewer_LQrgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces StepRel, an LLM-based approach to relational feature engineering that reduces the need to retain an agent's complete interaction history. The agent explores a relational database, generates SQL programs to construct predictive features, and iteratively refines these programs based on validation feedback from a downstream model.

The key observation is that effective feature search does not necessarily require preserving the entire interaction history. Instead, StepRel maintains three components outside the conversation: (1) reusable database knowledge, (2) the current best-performing SQL program, and (3) evidence from previous experiments, including unsuccessful attempts. The agent proposes local modifications to the stored program and periodically reconstructs its context from these components, avoiding repeated processing of the full interaction history.

Experiments on 18 RelBench tasks (10 binary classification and 8 regression), using four LLM backbones and comparisons with RelAgent and ReFuGe, demonstrate promising results. StepRel maintains or improves predictive performance while substantially reducing context length and token consumption.

Critical strengths
  1. Practical and relevant problem. Reducing the context and token costs of LLM agents for relational feature engineering while maintaining or improving predictive performance is an important problem. The proposed separation of persistent search state from interaction history is a reasonable and useful approach.

  2. Broad empirical evaluation. The evaluation covers 18 relational prediction tasks and four LLM backbones from three model families (DeepSeek, Qwen, and GPT). The comparison with RelAgent uses a shared evaluation and search protocol, providing a meaningful basis for assessing the approach.

  3. Substantial efficiency improvements. StepRel reduces prompt length by approximately 55% and cumulative token consumption by 33% compared with RelAgent on classification tasks using DeepSeek at 24 scored trials. These savings are achieved while maintaining competitive, and often improved, predictive performance. The reported reductions in estimated API costs further support the practical benefits.

Critical weaknesses
  1. Limited isolation of the individual contributions. The main comparison evaluates the complete StepRel system, which combines reusable database knowledge, local SQL program updates, and periodic context reconstruction. Although the paper provides targeted comparisons for database knowledge reuse and local program updates, the individual contribution of context reconstruction and the evidence-selection strategy is less clear. A more systematic ablation study would help establish which components are responsible for the improvements in predictive performance and efficiency.

  2. Potential limitations of single-incumbent search. StepRel maintains the best-performing SQL program as the starting point for subsequent modifications. Although information about unsuccessful and alternative candidates is preserved, these alternatives are not maintained as independent search branches. This may limit exploration of the feature space, particularly when improvements require coordinated changes to multiple feature blocks or temporarily accepting lower-performing candidates. While theoretical guarantees on search quality may be difficult to establish, additional experiments comparing single-incumbent search with broader exploration strategies would help assess the impact of this design choice.

  3. Limited methodological novelty. External agent memory, checkpointing, context compression, and incremental program editing have been explored in previous work. The main contribution appears to be their integration into relational feature search, rather than a new search or memory-management technique. The paper could more clearly articulate what is methodologically new and how the proposed design differs from existing approaches.

  4. Limited comparison with alternative memory-management strategies. The paper demonstrates improvements over RelAgent, which retains the full interaction history, but does not establish whether StepRel's specific state representation and evidence-selection strategy is better than simpler conversation summarization or existing agent-memory approaches. Such comparisons would help determine whether the gains come from the proposed design or primarily from avoiding the overhead of maintaining a full conversation history.

Questions for authors
  1. Can the authors isolate the effect of periodic context reconstruction and evidence selection from the other components of StepRel? In particular, how much of the observed improvement comes from shortening the conversation versus reusing database knowledge and making local SQL updates?

  2. How sensitive is the search quality to maintaining a single incumbent program? Would retaining multiple promising candidates or allowing multi-block modifications improve predictive performance, particularly for tasks requiring interactions among features?

  3. How does StepRel compare with simpler context-management strategies, such as periodically summarizing the interaction history, or with existing agent-memory methods under the same search protocol?

4 comments
AuthorsOpenReview ↗

(1/2) Isolating component contributions

Thank you for recognizing the value of separating persistent search state from interaction history, our broad evaluation, and the efficiency improvements. Your questions motivated additional component and history-management comparisons.

Q1/W1: What comes from shortening history versus knowledge reuse and local SQL updates?

We expanded the ablations to isolate knowledge reuse, local SQL updates, history clearing, and evidence selection. The core comparison covers six tasks across four backbones (three classification and three regression tasks across four databases, selected for coverage and short runtimes).

StepRel maintains database knowledge (K), the exact incumbent program (P), and cumulative structured search evidence (E). We test five controls:

  • E + full history: retain all interactions and K/P/E updates, disabling history clearing to isolate this part of context reconstruction.
  • Recent evaluation records (E): replace E with recent trial results, changes, and existing diagnostics, without the surrounding dialogue.
  • Recent interaction history (E): replace E with recent complete model/tool exchanges within a length budget. Neither recent-history control carries forward the previous E. Both retain K/P, local updates, and the reconstruction schedule.
  • No precomputed brief (K): remove upfront database knowledge while allowing online inspection.
  • Full SQL output (P): output complete candidate SQL instead of local edits, retaining the exact incumbent and single-block restriction.

Cells show mean test rank / total tokens (StepRel=100%), both lower being better. Ranks average task ranks after seed averaging. Costs sum task-mean brief and search input/output tokens. Overall weights backbones equally.

MethodDeepSeekGPTQwen3.6CoderOverall
Full StepRel3.67 / 100.0%2.67 / 100.0%1.83 / 100.0%3.33 / 100.0%2.88 / 100.0%
E + full history4.17 / 55.3%2.50 / 175.0%4.67 / 113.6%3.83 / 205.4%3.79 / 137.3%
Recent evaluation records (E)3.00 / 93.3%2.17 / 87.9%3.67 / 92.2%2.67 / 168.1%2.88 / 110.4%
Recent interaction history (E)3.50 / 84.6%4.83 / 146.1%3.00 / 100.3%3.67 / 144.6%3.75 / 118.9%
No precomputed brief (K)3.17 / 112.0%4.00 / 73.2%2.83 / 124.8%4.00 / 218.2%3.50 / 132.1%
Full SQL output (P)3.50 / 193.1%4.83 / 230.5%5.00 / 195.2%3.50 / 249.7%4.21 / 217.1%

Regarding “shortening the conversation versus reusing database knowledge and making local SQL updates”: Local updates provide consistent token savings across all four backbones. Knowledge reuse improves overall predictive rank and efficiency, although its upfront cost is not always recovered. Thus, shorter history alone does not explain the gains.

Regarding “isolate the effect of periodic context reconstruction and evidence selection”: The history controls retain K/P and local updates, separating these effects from knowledge reuse and the SQL output interface.

  • Context reconstruction: Clearing history while maintaining the same K/P/E updates improves overall rank and token cost. It saves tokens on GPT, Qwen3.6, and Coder, while retaining history is cheaper on DeepSeek. This isolates the history-clearing part of reconstruction.
  • Evidence selection: Full StepRel improves the overall performance-cost balance over recent interactions. Recent evaluation records match its overall rank, with higher overall cost driven by Coder. They remain a strong alternative, ranking better on three backbones, while StepRel ranks better on Qwen3.6.

Across these six methods, full StepRel achieves the joint-best overall predictive rank and the lowest overall normalized token cost. The effects vary by backbone, but the controls show contributions from both reducing history and organizing persistent search state. We will incorporate these results to clarify the individual design choices.

AuthorsOpenReview ↗

(2/2) Search alternatives and periodic summaries

Q2/Q3, W2/W4: How do broader search and periodic history summaries compare with StepRel?**

We compare the proposed alternatives on five common tasks with GPT and Qwen3.6 under the same trial budget. For Q2, three controls output complete SQL to separate search restrictions from the output interface:

  • Single-block reference: modify one block of the current incumbent.
  • Alternative parents: also modify one block of a retained alternative program.
  • Multi-block updates: allow several blocks to change relative to the incumbent.

For Q3, Periodic history summaries replaces structured E with a rolling summary of the previous summary and new interactions. It retains StepRel's database knowledge, exact incumbent, local updates, and reconstruction schedule. Full StepRel uses local edits and structured E.

Ranks compare all five methods within each task after seed averaging. Overall rank averages the ten task-backbone ranks. Mean total tokens (millions) includes the brief, search, and any summary generation, with tasks weighted equally.

MethodGPT rank ↓GPT tokens (M) ↓Qwen3.6 rank ↓Qwen3.6 tokens (M) ↓Overall rank ↓
Full StepRel3.200.6202.401.9472.80
Single-block reference3.201.4043.603.9213.40
Alternative parents2.201.5703.604.0882.90
Multi-block updates3.200.7062.602.9512.90
Periodic history summaries3.200.6452.801.9293.00

For Q2: Broader program updates can help in individual settings, but StepRel offers a better overall performance-cost balance. Alternative-parent and multi-block search have slightly worse overall ranks than StepRel and higher token costs on both backbones. Alternative-parent search nevertheless ranks best on GPT, and multi-block updates improve both classification and regression averages on Qwen relative to the single-block reference.

The cost comparison also reflects the update interface: these controls regenerate complete SQL, including unchanged blocks, while StepRel outputs local edits. The single-block reference shows that this interface contributes substantially to the cost gap, so the additional cost should not be attributed solely to broader exploration.

Single-incumbent search can still discover useful features through successive refinements. In an original DeepSeek user-churn run, a shopping-day-of-week feature received high importance but did not improve validation, so its candidate was rejected. After context reconstruction, the agent used retained block diagnostics to explore how purchasing was spread across dates. Adding distinct transaction dates and their ratio to transaction count improved validation AUC from 0.7058 to 0.7084. Extending the same block with active weeks and months raised it to 0.7086, while preserving the other seven blocks. This illustrates productive exploration from a single incumbent, without ruling out additional benefits from retaining alternative branches.

For Q3, with the database brief, exact program state, local edits, and reconstruction schedule held fixed, structured E achieves a slightly better overall predictive rank than periodic summaries, with similar token cost. The gain is modest, and periodic summaries remain a competitive alternative within StepRel.

Thank you again for your thoughtful review and for recognizing the practical value of separating persistent search state from interaction history, the breadth of our evaluation, and the efficiency improvements. Your questions helped us clarify the contributions of individual components and examine alternative search and memory strategies more thoroughly.

I wish to thank the authors for their response. I am maintaining my score which is already positive.

Thank you for considering our response and for your positive assessment! Your constructive questions helped us clarify the component contributions and explore alternative search and memory strategies. We will incorporate these findings into the revision to make the strengths and limitations of our approach clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.