- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes StepRel, a method for reducing context use in LLM-based relational feature search. It stores database knowledge, the current SQL program, and past search results outside the conversation. The agent updates individual SQL blocks and periodically rebuilds its context using the stored information. The goal is to maintain predictive performance without retaining the full interaction history.
Critical strengths
- The paper addresses the growing context cost of LLM agents that repeatedly generate and evaluate SQL programs. Separating database knowledge, the current program, and search evidence is a practical way to preserve useful information while reducing repeated context processing.
- The paper evaluates 18 relational prediction tasks across four LLM backbones. It compares predictive performance, stability, and token usage against existing methods. The experiments show that StepRel can maintain competitive predictive performance while substantially reducing context length.
Critical weaknesses
-
The method mainly combines existing ideas such as external state storage, local program updates, and context reconstruction. Appendix E evaluates database knowledge reuse and local updates, but does not isolate evidence selection or context reconstruction. This leaves the added value of the proposed state-management design unclear.
-
Although StepRel reduces context length, cumulative token savings vary across backbones and evaluation conditions. Appendix D shows that the benefits depend on brief-construction costs, search depth, and unusually long tool interactions. This limits the generality of the efficiency claims.
Questions for authors
-
Does the proposed evidence-selection strategy provide measurable benefits over a simpler baseline that uses the same database brief, current SQL program, and reconstruction schedule, but includes only recent evaluation results in the reconstructed context?
-
What is the effect of disabling periodic context reconstruction while keeping database knowledge, local program updates, and evidence selection unchanged?
-
At what search depths does StepRel achieve cumulative token savings across different backbones when the full database-brief construction cost is included for a single rollout?
3 comments
(2/2) End-to-end token costs
Q3/W2: At what search depth do savings cover the full database-brief cost?
We analyzed cumulative StepRel and RelAgent costs along their search trajectories, charging the full brief-construction cost to each StepRel rollout. This directly tests when search savings recover initialization and how the answer changes across backbones.
Costs use RelAgent=100%. Search change and brief overhead are measured relative to the same RelAgent total. The last column reports the first trial from which StepRel's mean cumulative cost stays below RelAgent through the observed endpoint. Coder brief costs are reconstructed offline.
| Backbone / task type | Search change | Brief overhead | StepRel total | Savings sustained from trial |
|---|---|---|---|---|
| DeepSeek / classification | -37.5% | +12.5% | 75.0% | 14 |
| DeepSeek / regression | -38.1% | +11.8% | 73.8% | 14 |
| GPT-Luna / classification | -31.7% | +4.6% | 72.9% | 19 |
| GPT-Luna / regression | -11.0% | +6.1% | 95.1% | 24 |
| Qwen3.6* / classification | -20.8% | +10.8% | 90.0% | 17 |
| Qwen3.6* / regression | -17.0% | +9.3% | 92.4% | 17 |
| Qwen3-Coder* / classification | -2.6% | +12.0% | 109.4% | Not observed |
| Qwen3-Coder* / regression | -41.0% | +5.2% | 64.1% | 3 |
Total tokens include cumulative input and output from search plus the full brief. Prompt length measures one call's input. Each curve uses the same matched run pairs throughout. A trial here means a completed candidate evaluation with a valid score.
Q3: When does StepRel become cheaper? Sustained savings begin at 14 trials for DeepSeek, 19–24 for GPT-Luna, and 17 for Qwen3.6. Coder regression recovers the brief cost by trial 3. Coder classification remains more expensive at the endpoint because its small search saving cannot cover brief generation. These are aggregate crossing points, not guarantees for every task.
(*) Both agents exhibited extreme loops with Qwen3.6, including 2,273 repetitions of a schema read by StepRel and 504 repetitions of a query by RelAgent. Coder also repeatedly requested tools after budget exhaustion despite explicit refusals, consuming tokens without new database evidence. Tool errors or refusals allow repair or abandonment without terminating the run, so these loops can continue. For starred results, we remove anomalous runs and their matched counterparts from the main-search comparison.
We thank the reviewer again for recognizing the practical motivation and breadth of the evaluation. Your questions prompted us to isolate the contribution of state management and examine when its efficiency benefits arise. We hope these additional experiments clarify both the value and the limits of the design and provide useful guidance for relational feature-search systems.
Thank you again for your constructive questions. We added controlled comparisons of evidence selection and history clearing, together with a fuller cost analysis including brief construction. We would appreciate any thoughts on whether these additions address your concerns and would be happy to clarify any remaining points.
(1/2) Evidence selection and context reconstruction
Thank you for recognizing the practical value of separating database knowledge, executable programs, and search evidence, and for your questions about state management and token savings.
Q1–Q2/W1: Isolating evidence selection and history clearing.
StepRel maintains database knowledge (K), the incumbent SQL program (P), and cumulative search evidence (E). We expanded the ablations to address your questions directly:
We evaluate all six methods on six tasks across four backbones (three classification and three regression tasks across four databases, selected for coverage and short runtimes).
Cells show mean test rank / total tokens (StepRel=100%), both lower being better. Ranks average task ranks after seed averaging. Costs sum task-mean brief and search input/output tokens. Overall weights backbones equally.
Q1: Benefit over recent evaluation results. StepRel matches this baseline's overall predictive rank while using fewer tokens overall. Benefits vary across backbones, with the predictive advantage on Qwen and the aggregate cost advantage driven by Coder. The additional recent-interaction control has a weaker overall performance–cost balance.
Q2: Effect of retaining the full history. Clearing old interactions improves overall predictive rank and reduces overall token cost. It saves tokens on GPT, Qwen, and Coder, while retained history is cheaper on DeepSeek.
W1: Contribution of the complete design. The additional controls show that the precomputed brief improves overall rank and efficiency, although its upfront cost is harder to recover in some settings. Local edits substantially reduce tokens on every backbone and improve overall rank compared with full SQL output. Across all six methods, full StepRel achieves the joint-best overall predictive rank with the lowest overall normalized token cost. These results clarify the practical contribution of the components while retaining the backbone-dependent tradeoffs.