- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes P-UTL, a memory management method for long-horizon Text-to-SQL. The core idea is to maintain a fixed-size memory pool of entries, and use a lightweight MLP to predict each entry's future utility. When the pool is full, only the top-B entries are retained. The MLP is trained offline on SQL execution data: removing an entry and re-running the query to see if accuracy changes. Experiments on synthetic scenarios show P-UTL outperforms baselines including LRU and LLM-based manager, especially under high memory pressure.
Critical strengths
S1. Long-horizon conversation memory is an important and underexplored problem. Real-world multi-turn Text-to-SQL systems frequently encounter situations where early context (e.g., table definitions, business rules) must be recalled dozens of turns later. Existing methods either keep everything (costly) or use generic heuristics like recency. P-UTL addresses this gap with a principled approach: learning which entries actually matter based on downstream task performance. Even if the proposed method has limitations, the problem formulation itself advances the field.
Critical weaknesses
W1. Single-pass eviction has an irreversible information loss problem with no mitigation. Once an entry is evicted, it is permanently gone and cannot be retrieved even if later queries depend on it. This is an architectural flaw and fundamentally limits the method's applicability to truly long conversations where early context may become relevant again after many turns.
W2. The validity of the evaluation dataset is questionable. The entire benchmark is synthetically constructed: recall schedules follow hand-picked distributions (Normal-35, Poisson-40). There is no evidence that these distributions reflect real user behavior in long-horizon Text-to-SQL conversations. The paper does not report how conventions are phrased (template vs. free-form). Without validation on real or more diverse data, the reported improvements may be artifacts of the synthetic setup rather than genuine advances in memory management.
Questions for authors
Q1. Have you tested P-UTL in the adversarial setting where new concepts are continuously introduced, their total number far exceeds the memory budget B, and queries uniformly sample from all historical concepts? If so, what were the results? If not, can you provide a theoretical analysis of how P-UTL would behave in this case?
Q2. How were the conventions (definitions) generated? If they follow a template (e.g., "define X as Y"), does the MLP simply learn template-specific cues? Do you have any results on free-form, human-written definitions, or cross-database generalization?
Q3. Are the training and test datasets generated from the same synthetic pipeline? If so, could the MLP be overfitting to distributional regularities of that pipeline rather than learning a general notion of utility? Have you validated P-UTL on data from a different generation process, such as different recall distributions or real human conversations? Without such cross-distribution validation, how can we rule out that P-UTL's advantage over LRU is largely an artifact of overfitting to the specific synthetic distribution used in both training and evaluation?
2 comments
Dear Reviewer,
While the rebuttal is ending soon, we looking forward to your response to check whether our rebuttal addressed your concerns. We would be happy to make clarifications if you had concerns unresolved.
Best,
Submission 22100 Authors
Response to Reviewer EGyU
We thank the reviewer for their thoughtful feedback. Below, we clarify the convention-generation procedure and present additional experiments addressing irreversible eviction and generalization beyond fixed definition templates. We welcome the reviewer’s further feedback on these responses.
1. Irreversible eviction under continuous concept arrivals
The reviewer is correct that an evicted entry cannot be recovered from the bounded store, and we do not claim that P-UTL can recall every historical concept indefinitely. When future requests may concern any past concept, a finite memory pool cannot guarantee perfect recall. The fair comparison here is how different policies perform under the same memory constraint.
To evaluate the proposed adversarial setting, we construct 100-round trajectories in which new definitions and their associated warm-up turns are introduced densely throughout the interaction (see Figure 3 for the definitions of these turn types). We randomly insert 10 callbacks between turns 26 and 100, with each callback referring to a definition randomly selected from all preceding definitions. We generate three such trajectories for evaluation on the
department_storedatabase.P-UTL outperforms E-LLM and LRU even when the number of definitions exceeds the memory budget and callbacks sample the full available history. These results demonstrate an advantage beyond the original recurrence schedule.
2. How the submitted conventions are generated
The convention workloads in the submitted paper use a fixed template. Each definition specifies that a named term denotes rows satisfying a numeric predicate and includes an initial COUNT query. This is followed by three warm-up queries, AVG, COUNT(DISTINCT), and MAX, in shuffled order, and a delayed SUM callback. An example is shown below:
To examine generalization beyond this template, we conduct an additional experiment using free-form definitions, as described in the next section.
3. Generalization to free-form definitions
To test whether P-UTL’s performance depends on the fixed definition wording, we replace the templated definitions with LLM-generated free-form descriptions of the same executable predicates and check their semantic fidelity. We keep the initial questions, warm-up queries, callback operations, and recurrence patterns unchanged. An example is:
We regenerate the workload using Poisson distribution (λ=40), normal spacing, and 30 callbacks. We evaluate the pretrained P-UTL-FULL predictor directly, without retraining.
P-UTL-FULL handles free-form definitions without retraining, outperforming E-LLM and LRU by a large margin. Since these definition forms are absent from its training data, the results provide evidence that P-UTL generalizes beyond the specific definition template used in training.