acceptodds
Under review as a conference paper at ICLR 2027

Learning Task-State Abstractions for Long-Horizon Database Agents

Abstract

Tool-using database agents must track what they know, what they have tried, and which evidence should guide the next action. We study how structured task-state representations support this sequential decision problem. TSADB combines schema topology, query intent, execution traces, and error semantics into a learned state consumed by a lightweight PPO policy; a frozen language model generates tool arguments. On BIRD-SQL and Spider 2.0-Lite, the system improves execution accuracy by 10.0 and 10.4 percentage points over the strongest matched baseline in each main comparison. A separate training-budget-matched comparison with RL-ReAct yields a 7.9-point BIRD gain. A state-by-reward factorial shows complementary contributions: flat-history encoding reduces accuracy by 9.1 points with the progress reward, while removing that reward costs 10.5 points with the learned state. The state effect grows to 18.5 points on tasks requiring at least 16 calls. Across policy classes, supervised control also benefits from the representation, and PPO adds 13.1 points over supervised control on long tasks in a shared 800-task evaluation. The full system retains 74% of its accuracy under combined schema perturbations. Experiments further examine official conversational interaction, transfer with limited error-projection adaptation, termination, and reward construction. These results support learning compact, structured states for long-horizon database control.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, state abstraction, tool-using agents, database agents, long-horizon decision making

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KG2vOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TSADB, a database agent that uses schema information, query intent, execution history, and errors to build a state for choosing tools. A frozen LLM then generates the tool arguments. The authors test it on BIRD, Spider 2.0-Lite, and DS-Bench, alongside studies of rewards, controllers, robustness, and transfer to help explain the gains. The results show gains under matched interfaces and budgets, with the largest reported benefits on tasks requiring many tool calls.

Critical strengths
  1. The experiments separate the effects of the state representation, reward, and controller to make it much easier to see where the gains come from.

  2. TSADB beats RL-ReAct by 7.9 points on BIRD with the same training budget. The paper also makes clear how adapting the baseline interfaces affects their results.

Critical weaknesses
  1. The paper does not specify how the reward relevance projection is trained during warmup, including its objective and update rule. This omission limits reproducibility of the progress-reward component.
  2. The recovery procedure is not clearly distinguished from the standard interaction loop, making the reported 6.7 point ablation gain difficult to interpret. The tool masks and validation rules also require a more complete description.
Questions for authors
  1. Please specify the training objective and update rule for the reward relevance projection during warmup. Is this phase included in the 2,000 episode training budget?
  2. Please clarify how the recovery procedure differs from the standard interaction loop and what its ablation removes. Please also provide the complete tool-masking and validation rules, including the treatment of valid empty results.
  3. Please describe the task and database splits used for adaptation and evaluation, including their separation from model selection. What data and training budget are used for the full-retraining reference?
Reviewer_anQ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the challenge of tool selection in long-horizon database tasks, where an agent must iteratively examine the database schema, execute queries, and determine its next steps based on the results or error messages. The authors propose TSADB (Task-State Abstraction for DataBase agents), which employs Task-State Abstraction (TSA) to aggregate information—including the database schema, user intent, execution history, and error logs—to facilitate tool selection via a lightweight policy. The underlying language model remains frozen and is tasked solely with generating specific parameters. During training, a progress reward is utilized to encourage the agent to incrementally identify relevant database fields. TSADB achieved execution accuracies of 82.0% on BIRD-SQL and 44.6% on Spider 2.0-Lite, outperforming the strongest comparable baselines. The benefits of structured state representation are particularly pronounced in tasks requiring multiple tool invocations. The paper concludes that enabling agents to effectively organize evidence gathered during interactions improves both decision-making and task completion rates in long-horizon scenarios.

Critical strengths

The paper concretizes the challenges faced by long-horizon, database-augmented agents—specifically, the problem of how to extract states from an ever-growing interaction history that are useful for subsequent decision-making—which is of significant practical relevance.

The proposed method demonstrates clear improvements over standard history-encoding techniques in long-horizon tasks, while incurring relatively lower overhead.

Critical weaknesses

The RL controls (RL-ReAct, RL-MAC-SQL) share the PPO procedure, composite reward, and episode budget, but under a frozen GPT-5 backbone it is unclear what parameters PPO actually optimizes—certainly not the LLM itself. The Appendix G fine-tuning studies also do not fill this gap: one LoRA-tunes the argument generator with the external policy kept (Table 11), and the other LoRA-tunes a Llama-3.2-3B policy head that classifies over the pre-computed state vector s_t (Table 12, +0.8 EX)—neither constitutes end-to-end post-training of a generative LLM agent under the same reward. Without such a baseline, the case for a separate lightweight policy over direct LLM post-training remains unestablished.

The roles of "structured information" and the "independent policy network" have not been sufficiently disentangled. Although the paper compares TSA with a "flat history" approach, it lacks a direct control: providing the same structured information to the LLM as text and allowing the LLM to select tools autonomously.

The scope of the method is constrained by fixed design choices. The current policy is limited to selecting from five specific tools, with an execution history window fixed at 10 entries and an inference limit of 20 tool calls. While these parameters prove effective for existing benchmarks, there is insufficient evidence to suggest they would remain applicable to longer-horizon tasks or workflows involving different sets of tools.

All ablations in Table 12 are one-at-a-time removals; no combined ablations or interaction analyses are reported. The authors themselves note that component effects "depend on the remaining configuration," so the measured 9.1-point state effect may be inflated or deflated by the specific workflow (masking, adaptive loop, validator) it is evaluated in. Additive-vs-overlapping contributions of the major components remain unknown.

Overall, the methodological contributions of this work are limited, and it is best characterized as an incremental study.

Questions for authors

Why train a separate tool-use strategy rather than simply performing post-training on the generative LLM itself? While training a tool-use strategy might be less costly, an LLM-based approach could potentially offer superior generalization capabilities. If this method lacks a clear advantage, the contribution of this work becomes even more limited.

Does the performance gain stem from the way TSA organizes information, or from training a separate strategy network? Could the same structured information be provided to an LLM in text format—allowing it to select directly from the five available tools—and then compared against TSADB?

Why is PPO strictly necessary? The supervised learning baseline in the paper is weaker than PPO; however, if one were to match the number of trajectories, environment interactions, and computational costs—while also scaling up SFT data or employing iterative behavioral cloning—would PPO’s advantage still hold?

It would be best to include comparisons with work that involves post-training LLMs directly—for example, MTIR-SQL (Multi-turn Tool-Integrated Reasoning Reinforcement Learning for Text-to-SQL) and AGRO-SQL (Agentic Group-Relative Optimization with High-Fidelity Data Synthesis).

4 comments
AuthorsOpenReview ↗

Respond for Reviewer anQ5 [Part 1/2]

Thank you for proposing a controller that uses the structured information directly as text (W2 and Q2).

Structured-input text router: At each step, we serialize the four information sources used by TSA: revealed schema and foreign-key relations, the current request/intent, the last ten action-observation summaries, and cumulative twelve-category error counts plus the latest error. Frozen GPT-5 receives this structured text and the same tool-validity mask, selects one of the five tools, and then generates arguments.

The comparison uses the same tool interface and a 20-action allowance covering tool executions and argument-regeneration retries. Additional LLM routing prompts are charged to the 120-second wall-clock limit and token/latency measurements; they do not consume tool-action slots. This preserves the action horizon while exposing the text router's inference cost.

Controller with frozen GPT-5Full-dev BIRD EX (%)16+ EX (%)Total ms/step
ReFoRCE72.038.41,050
Structured-input text router74.446.51,410
Supervised MLP + TSA73.647.2846
PPO MLP + TSA82.060.3846

Providing structured information to the text router improves over ReFoRCE by 2.4 points overall and 8.1 on long tasks. The supervised TSA controller reaches similar accuracy with 40% lower total per-step latency. PPO-based TSADB exceeds the text router by 7.6 points overall and 13.8 on long tasks, also at 40% lower total per-step latency.

This result supports the complete learned-control design. The policy ladder provides the comparison within the TSA architecture: PPO versus supervised control improves long-task EX by 13.1 points. We have presented the complementary comparisons in Section 5.5 (pp. 8–9) and Table 5 (p. 9). Appendix B.7 (p. 23) and Table 12 (p. 23) specify the serialization and budget accounting.

State by workflow scaffold (W4): We jointly removed tool masking, the adaptive loop, and argument validation, retaining the specified training reward:

ConfigurationFull-dev BIRD EX (%)16+ EX (%)
Flat state, scaffold removed63.831.8
TSA, scaffold removed72.446.9
Flat state, full scaffold72.941.8
TSA, full scaffold82.060.3

The TSA effect is 8.6 points without the scaffold and 9.1 with it. On long tasks, the effects are 15.1 and 18.5 points. The corresponding differences-in-differences are +0.5 overall and +3.4 on long tasks. State organization improves completion in both scaffold conditions, with a larger interaction on long tasks. We have added this joint intervention in Table 14 (p. 24), alongside the individual ablations in Table 22 (p. 31).

Trace window and inference horizon (W3): With trace-window sizes K=5,10,15,20K=5,10,15,20, overall BIRD EX is 79.8%, 82.0%, 82.2%, and 82.1%. Gains beyond ten pairs are small in this sweep; cumulative schema and error summaries remain available separately from the trace window.

On the same 85 long tasks, TSADB achieves 44.7%, 60.3%, and 63.5% EX with inference caps H=15,20,30H=15,20,30. ReFoRCE reaches 38.4% at H=20H=20 and 39.8% at H=30H=30. These results distinguish the selected twenty-call operating point from performance with a larger inference allowance. The long-task group is defined by canonical reference horizon, independently of the evaluated cap.

Interface scope: The submitted experiments also cover Python execution through the DS-Bench interface, where the gain is 9.9 percentage points in TSR, and official c-Interact conversation, where the gain over RL-ReAct is 7.6 points. Larger and dynamically changing action spaces remain a useful extension of the five-tool controller. We have collected the window and horizon sweeps in Table 15 (p. 24) and discuss the interface scope in Section 6 (p. 10).

AuthorsOpenReview ↗

Respond for Reviewer anQ5 [Part 2/2]

Iterative behavior cloning under the same interaction budget (Q3): We ran four rounds of trajectory collection and cross-entropy updates, allocating 500 rollouts per round for 2,000 episodes in total. Each round filters for successful trajectories before updating the policy.

Training methodFull-dev BIRD EX (%)16+ EX (%)
Single-round behavior cloning76.549.6
Four-round iterative behavior cloning78.352.9
PPO82.060.3

Iterating BC improves overall EX by 1.8 points, while PPO retains advantages of 3.7 points overall and 7.4 on long tasks. PPO uses stepwise rewards and value estimates from trajectories including unsuccessful completions; this outcome-filtered BC uses successful trajectories and applies cross-entropy to their steps. This experiment matches rollout collection across methods and compares their respective update procedures. The results favor PPO under this interaction budget. We have added the iterative-BC protocol in Appendix B.7 (p. 23) and its result in Table 5 (p. 9).

Parameters optimized by RL-ReAct and RL-MAC-SQL (W1 and Q1): With GPT-5 frozen, PPO updates their parameter-matched flat-history encoders and two-layer routing/value heads. GPT-5 supplies verbal thoughts and tool arguments. The trainable components total approximately 0.5M parameters, including about 0.3M for the learned encoding components and 0.2M for routing/value heads; pretrained mpnet and GPT-5 weights remain frozen.

End-to-end multi-turn RL (Q4): Using Llama-3.1-8B, we also trained an autoregressive policy that generates both tool selections and arguments under the composite reward and 2,000-episode budget:

MethodTrainable parametersGPU-hoursFull-dev EX (%)16+ EX (%)
End-to-end multi-turn PPO21M LoRA21063.534.1
End-to-end group-relative RL21M LoRA18564.235.3
Decoupled TSADB, frozen argument LLM0.5M7266.943.5
TSADB with argument LoRA0.5M + 21M9669.847.1

The end-to-end runs use LoRA rank 16 and eight A100 GPUs. At this budget, decoupled control gives higher accuracy at lower training cost; argument-model tuning provides a further improvement. These comparisons complement the API-model setting, where the argument model remains frozen.

We have added these in-environment end-to-end controls in Appendix G.1 (pp. 29–30), with results in Table 21 (p. 30). Section 2 (p. 2) distinguishes their algorithms from MTIR-SQL, which extends GRPO for multi-turn tool-integrated reasoning, and AGRO-SQL, which combines data synthesis, cold-start training, and group-relative optimization.

AuthorsOpenReview ↗

Followup Response for Reviewer anQ5 (Clarifying the Evidence for Structured State Organization and Decoupled Policy Learning)

Sincerely thank you again for the thoughtful and constructive review. We would like to further clarify how the controlled experiments address two central questions in your review: whether the gains come from organizing task evidence or merely training a separate controller (W2/Q2), and whether decoupled control provides an advantage over direct generative-LLM post-training (W1/Q1).

1. Distinguishing information access, state organization, and policy learning

Three complementary comparisons on the same 1,534-task BIRD development split help distinguish these factors:

1) Information access vs. representation: Under the same PPO procedure, reward, policy-head architecture, and workflow scaffold, a memory-matched flat encoder receives the same revealed schema information, cumulative error memory, and ten-pair interaction history as TSA. It reaches 74.8% EX, compared with 82.0% for TSA. A PPO controller using observable hand-built state features reaches 75.6%. These controls directly test the value of learned state organization beyond memory availability (Table 5, p. 9; Appendix B.6).

2) Structured text vs. learned control: Providing the same categories of observable information as text to frozen GPT-5 for tool routing yields 74.4% EX, comparable to supervised TSA control at 73.6%, but below PPO-trained TSA at 82.0%. The text router also requires 1,410 versus 846 ms/step (Table 13, p. 23). Since the text-router and PPO comparison changes both policy architecture and training, we interpret it alongside—not as a substitute for—the within-framework controls.

3) Sequential optimization: With TSA fixed, supervised control reaches 73.6%, four-round iterative BC reaches 78.3%, and PPO reaches 82.0%. BC and PPO use the same 2,000 rollout-episode budget, although their update procedures and computational costs differ (Table 4 and Appendix B.7).

Together, these results distinguish the contribution of organizing observed evidence from that of optimizing the tool-selection strategy.

2. Direct comparison with generative-LLM post-training

To address your concern about whether a separate controller is preferable to updating a generative agent directly, we conducted same-backbone experiments with Llama-3.1-8B under the shared environment, composite reward, and 2,000-episode budget (Appendix G.1, Table 22, p. 30).

End-to-end multi-turn PPO and group-relative RL, which generate both tool selections and arguments autoregressively, achieve 63.5% and 64.2% BIRD EX at 210 and 185 GPU-hours, respectively. Decoupled TSADB with the same frozen Llama backbone achieves 66.9% at 72 GPU-hours; additionally tuning its argument generator with LoRA increases EX to 69.8% at 96 GPU-hours.

Thus, in this controlled setting, separating tool-selection learning from argument generation provides both an accuracy advantage and a more favorable training-cost trade-off. These in-environment controls directly test the architectural choice; they are not claimed to reproduce the complete training and data-synthesis recipes of MTIR-SQL or AGRO-SQL.

3. What the combined controls establish

The State × Scaffold factorial further shows that TSA retains an 8.6-point advantage after removing masking, recovery/stopping, and argument validation (Table 15, p. 24). The State × Progress-Reward interaction increases from +2.2 points overall to +6.2 points on 16+-call tasks (Table 3), indicating stronger complementarity on extended trajectories.

The central empirical finding is that structured organization of accumulated evidence contributes beyond simply providing more history, while compact learned control offers an effective alternative to direct generative policy training under the evaluated task and compute budgets. This is the distinction our experiments aim to establish beyond complete-system leaderboard gains.

For clarity, the latest revised manuscript also adds the detailed RL-baseline trainable-module specification in Table 11 (p. 22), with GPT-5/mpnet frozen and the controller and reward-projection parameter budgets reported separately.

We hope this synthesis clarifies the methodological contribution and how the different controls support it. We welcome any further feedback during the discussion period.

AuthorsOpenReview ↗

Updated Manuscript: Additional Analyses Now Documented in Appendix L

Dear Reviewer anQ5,

We would like to briefly note that the two additional analyses presented in our recent follow-up have now been incorporated into the latest revised manuscript, with their experimental configurations and results documented in Appendix L (pp. 35–36):

Appendix L.1, Table 24 (p. 35): The representation bridge from flat history through memory-matched and four-branch MLP-only encoders to full TSA, including the 16+-call results and the distinction from individual encoder ablations.

Appendix L.2, Table 25 (p. 36): The extended backbone-to-tool interface adaptation audit covering all seven training-free baseline families across BIRD-SQL, Spider 2.0-Lite, and DS-Bench.

The principal benchmark results and existing main-text table numbering remain unchanged.

We hope these additions make the new evidence easier to locate and assess during the remaining discussion and subsequent AC–reviewer deliberations. Thank you again for your consideration.

Reviewer_ko6EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TSADB, a controller for LLM database agents. Instead of letting the LLM work from raw history, a small learned module builds a "task state" from the schema seen so far, the question, the last 10 action/observation pairs and a histogram of 12 error types. The schema goes through a GCN, the question through a frozen mpnet encoder, and the recent trace through a small Transformer. An MLP reads that state and picks one of five tools, and a frozen GPT-5 then writes the arguments. The whole thing is trained with PPO on a mix of correctness, efficiency and a schema-coverage progress reward.

They report 82.0 EX on BIRD dev against 72.0 for the best baseline, and 44.6 on Spider 2.0-Lite against 34.2. They also run a state x progress-reward 2x2, a ladder of policy classes from random up to PPO, schema perturbations, and a few transfer experiments on SPARQL and BIRD-Interact. The main claim is that the structured state and the progress reward each contribute, and that the state matters more on long tasks needing 16 or more calls.

Critical strengths
  • The state, reward and policy class are varied separately. Table 3 and Table 4 show how much each of the three contributes, instead of only comparing the full system with other agents. The flat-state control keeps the same mask, budget and rewards, so the 9.1-point state effect inside TSADB is a controlled comparison, although the two states also differ in memory organization. This is the main reason the paper is more than a leaderboard result.

  • Baseline adaptation is documented. Appendix B reports six baseline families on BIRD in their original configuration, backbone-matched and tool-matched. The horizon bins come from gold solutions, not from agent behaviour. This is more transparent than a bare leaderboard, though the audit does not cover every baseline or the Spider and DS-Bench settings.

  • Useful reference points. The ladder includes a random policy, a heuristic router, behavior cloning, a contextual bandit and GRPO. Table 5 includes oracle termination, and a LoRA-tuned Llama-3.2-3B policy head replaces the MLP tool policy in a capacity study. These show where the gains sit instead of leaving the reader to guess.

  • Consistent gains across settings. TSADB beats the strongest baseline on BIRD, Spider 2.0-Lite and DS-Bench. On BIRD the appendix reports the improvement over the better of ReFoRCE and SDE-SQL-style on each of five backbones, from 7.7 to 10.0 points. The long-task gap is also large, 60.3 against 38.4 for ReFoRCE on the 16+ bucket.

  • Breadth of evaluation. Beyond the three main benchmarks there are schema perturbations, SPARQL transfer, official BIRD-Interact evaluation and an extension of the error taxonomy. The paper is also clear about which interaction protocol each number uses.

Critical weaknesses
  • The headline comparison mixes RL with the learned state.

    • TSADB is PPO-trained on gold rewards, and as far as I can tell the Table 1 baselines get no training, so a good part of the roughly 10-point gains is RL against no RL.
    • The RL-matched gap on BIRD is 7.9 over RL-ReAct.
    • On Spider 2.0-Lite the RL controls gain only about 1.4 to 2 points over their untrained versions, so RL can't be the whole story there. There is no accuracy ablation on Spider to say what is, and Table 3 only gives shuffle retention.
  • The rest of the scaffold may explain much of the gap.

    • Taking out the adaptive loop costs about 7 points, the mask about 6 (Table 12), and the stop rule about 4 (Table 5). The stop rule probably sits inside the loop, so I wouldn't add them, but next to the 9.1-point state effect they aren't negligible.
    • The paper never says whether RL-ReAct and RL-MAC-SQL get the mask, validator, recovery trigger or 0.95 stop rule.
    • To be fair, the state comparisons in Tables 3, 4 and 12 are controlled, and flat-history TSADB keeps the mask yet still lands below RL-ReAct, 72.9 against 74.1. That cuts against my guess, but I couldn't tell whether it also gets recovery and the stop rule.
  • The flat baseline isn't memory-matched.

    • It sees the last 8 observations, while TSA also carries a running schema graph and error histogram. The authors acknowledge this and move on, so some of the 18-point gap on long tasks could just be memory.
    • The GCN-to-MLP row in Table 12 helps a bit, but it explains under 3 points.
    • I found no learned policy on a plain hand-built feature state, such as tables seen, error counts, last tool and step count. I also never found out what the 61.4 heuristic router actually does.
  • Table 4 is hard to read.

    • The 800-task subset is described as stratified and keeping all 85 long tasks, but PPO-TSA at 82.0, GRPO at 79.6 and the supervised 73.6 all equal full-dev numbers from elsewhere.
    • The paper says full-dev and subset results are reported separately, and I can't find where.
Questions for authors
  1. Do RL-ReAct and RL-MAC-SQL get the same mask, validator, recovery and stop rule as TSADB? If not, what do they score when they do?
  2. Which Table 4 numbers come from the 800-task subset, and how was it sampled? PPO-TSA, GRPO and supervised all equal full-dev values.
  3. Section 5.1 trains on BIRD and Spider 1.0, but Figure 2 plots Spider 2.0-Lite against training episodes. Did any Spider 2.0-Lite tasks go into training, reward-weight selection, the 0.95 threshold or the validator? Where do the 200 validation tasks come from?
  4. How does PPO on a hand-built feature state, without the GCN and Transformer, compare?
8 comments
AuthorsOpenReview ↗

Respond for Reviewer ko6E [Part 1/2]

Thank you for asking how scaffolding and memory access affect the comparison (Q1 and Q4).

Scaffold parity: The submitted RL-ReAct and RL-MAC-SQL controls use the shared tool-validity mask and argument validator, while retaining their native termination loops. The flat-history TSADB control retains the full TSADB scaffold, including recovery and the 0.95 stopping rule. We additionally equipped both RL baselines with those recovery and stopping rules:

ConfigurationBIRD EX (%)16+ EX (%)
RL-ReAct, original loop74.142.4
RL-ReAct, full scaffold75.444.7
RL-MAC-SQL, original loop71.737.6
RL-MAC-SQL, full scaffold73.040.0
TSADB, full scaffold82.060.3

With these components matched, TSADB leads RL-ReAct by 6.6 points overall and 15.6 on long tasks. We have distinguished the original-loop and full-scaffold comparisons in Section 5.3 (p. 6), Table 5 (p. 9), and Table 11 (p. 22).

Memory-matched flat encoding: The original flat control uses eight recent observations, whereas TSA also maintains cumulative schema and error information. The new flat control receives all revealed schema elements, cumulative twelve-category error counts, and the same ten action-observation pairs as TSA, with a matched encoder parameter budget. It reaches 74.8% overall and 45.9% on long tasks, versus 72.9% and 41.8% for the original flat control. Matching memory access recovers 1.9 and 4.1 points, respectively; the remaining TSA gaps are 7.2 and 14.4 points.

Observable hand-built features: A further PPO control uses revealed table/column/foreign-key counts, inspection coverage within the observed table directory, cumulative and latest-error features, the last three tools and outcomes, normalized step count, and query-to-revealed-column similarity summaries. It reaches 75.6% overall and 48.2% on long tasks, below TSA by 6.4 and 12.1 points. Together, these controls support learning relational and sequential representations beyond providing persistent memory or scalar summaries.

RL-ReAct also generates verbal Thought summaries, whereas the original flat TSADB control encodes raw recent observations. This difference is relevant to interpreting their 74.1% versus 72.9% scores; the new memory-matched comparison addresses the broader difference in historical access directly.

For the 61.4% heuristic router, initialization and schema errors trigger inspection; diagnostic calls are followed by SQL execution when available. Other execution errors trigger error analysis. A successful query is optimized above 1.0 s execution time or 400 tokens, then validated; a score below 0.95 returns control to query revision. We have specified the router, rule ordering, and tool eligibility in Appendix A.3 (pp. 17–18); Appendix B.6 (p. 22) describes the memory and feature controls.

Training provenance and Spider 2.0-Lite ablations (Q3): The policy is trained on 300 BIRD tasks drawn from 55 databases and 200 classic Spider 1.0 tasks drawn from 120 databases. Model selection uses another 200 tasks: 120 BIRD tasks from 14 held-out databases and 80 Spider 1.0 tasks from 20 held-out databases. The 55/14 BIRD partition uses only the 69 official training databases; the Spider partition also uses its training release. Spider 2.0-Lite tasks and databases are excluded from policy training, relevance warmup, reward-weight selection, and validator/threshold design.

Figure 2 evaluates saved checkpoints on Spider 2.0-Lite during this training run. It shows transfer performance as training progresses on BIRD and classic Spider. We have stated the training sources in the caption of Figure 2 (p. 7) and the split details in Appendix A.5 (pp. 19–20).

The following execution accuracies complement the rounded retention values in Table 3:

ConfigurationUnperturbed EX (%)Full Shuffle EX (%)Retention (%)
TSA + PR44.633.074.0
TSA, no PR39.427.670.1
Flat history + PR36.120.957.9
Flat history, no PR33.517.853.1

With PR retained, replacing TSA with flat history costs 8.5 points on unperturbed tasks and 12.1 under Full Shuffle. With TSA retained, removing PR costs 5.2 and 5.4 points, respectively. These comparisons distinguish absolute completion accuracy from perturbation retention and show contributions from both the representation and progress feedback. We have added the absolute accuracies in Table 13 (p. 23).

AuthorsOpenReview ↗

Respond for Reviewer ko6E [Part 2/2]

Table 4 subset and full-development results (Q2): We found a table assembly error in the Overall EX column: the supervised TSA, GRPO-TSA, and PPO-TSA rows contained full-development scores, while the other rows contained 800-task scores. We have replaced the mixed column with two explicitly labeled columns in Table 4 (p. 8) and corrected the dependent differences in Section 5.5 (p. 8). The sampling allocation is specified in Appendix B.2 (p. 21).

Policy800-task EX (%)Full-dev EX (%)16+ EX (%)
Random valid action21.722.18.9
Heuristic finite-state router61.462.029.5
Supervised MLP, flat state67.968.438.2
Supervised MLP, TSA73.173.647.2
Contextual bandit, TSA75.275.846.0
Behavior cloning, TSA76.076.549.6
GRPO, TSA79.179.657.3
PPO, TSA81.482.060.3

The 800-task subset contains all 85 tasks in the 16+ bin, plus 151, 296, 167, and 101 tasks from the 1–3, 4–6, 7–10, and 11–15 bins. The latter allocation samples approximately 49.34% of the remaining 1,449 tasks, using NumPy's default_rng(seed=42). All policy rows use this same subset. Both evaluation sets contain exactly the same 85 long tasks, so the 16+ column is shared.

The corrected subset entries are 73.1, 79.1, and 81.4; their full-development counterparts are 73.6, 79.6, and 82.0. On the corrected subset, the supervised TSA gain over supervised flat control is 5.2 points, and PPO improves over supervised TSA by 8.3 points overall. The long-task differences remain 9.0 and 13.1 points, respectively. Full-development main results, including 82.0% for C-full, retain their original evaluation scope.

AuthorsOpenReview ↗

Additional Clarification on State–Scaffold Interaction and RL Training Controls

Sincerely thank you again for your careful review. We would like to add one comment on highlighting two methodological clarifications from the revised manuscript that were not explicitly connected to your concerns in our initial two-part response, particularly your observation that "the rest of the scaffold may explain much of the gap."

1. Separate recovery/stopping interventions and State × Scaffold factorial

1) Recovery v.s. stopping. You correctly noted that the stopping rule likely contributes to the adaptive-loop ablation and that these effects should not simply be added. We have now explicitly separated the interventions (Appendix A.3, p. 18):

  • Removing only recovery: 82.0% → 78.1% (−3.9 pp).
  • Removing only the 0.95 stopping rule: 82.0% → 78.4% (−3.6 pp).
  • Removing both (adaptive-loop ablation): 82.0% → 75.3% (−6.7 pp).

The joint 6.7-point loss is smaller than the 7.5-point sum of the individual losses, confirming that their contributions overlap rather than add independently (Tables 6 and 23).

2) Whether the state benefit depend on the scaffold. Beyond matching RL-ReAct's recovery and stopping rules, we also evaluated a within-framework State × Scaffold 2×2 factorial (Section 5.5; Appendix B.8, Table 15, p. 24). Scaffold removal jointly disables tool masking, the adaptive loop (including recovery and stopping), and argument validation while retaining the same PPO procedure and reward within each condition.

Scaffold ConfigurationFlat State EX (%)TSA EX (%)State Effect (Δ\Delta)Flat 16+ EX (%)TSA 16+ EX (%)16+ State Effect (Δ\Delta)
Full scaffold72.982.0+9.1 pp41.860.3+18.5 pp
Scaffold removed63.872.4+8.6 pp31.846.9+15.1 pp

Crucially, TSA retains an 8.6-point overall advantage and a 15.1-point advantage on the 85 longest tasks even without the scaffold. The state–scaffold interaction is +0.5 points overall and +3.4 points on long tasks. Thus, the measured representation benefit is not primarily explained by the surrounding scaffold, although the scaffold improves absolute performance and shows greater complementarity with the state on long tasks.

2. RL baseline training parity and updated manuscript references

Following a further clarification requested during discussion, the latest revised manuscript now explicitly specifies the trainable architectures of RL-ReAct, RL-MAC-SQL, and RL-DS-Agent (Section 5.3, p. 6; Figure 2 caption, p. 7; Appendix B.6, pp. 22–23).

All three keep GPT-5 and mpnet frozen while PPO updates a ~0.3M-parameter flat-history encoder and ~0.2M-parameter MLP policy/value heads. They share the composite reward, tool interface, validity mask, argument validator, and 2,000-episode training budget. The shared ~0.14M training-time relevance projections are accounted for separately.

We distinguish these training-matched workflow comparisons, which retain different native prompting procedures, from our within-framework representation controls, which hold the PPO procedure, policy-head architecture, reward, and scaffold fixed. In particular, the memory-matched flat encoder (74.8% EX) aligns cumulative information access, while the hand-built feature control (75.6% EX) tests whether coarse observable state summaries suffice (Table 5, p. 9; Appendix B.6).

For reference, the new RL architecture table is Table 11 (p. 22); the scaffold-alignment results are now Table 12 (p. 23), and the Spider 2.0-Lite accuracy ablations are Table 14 (p. 24).

We hope these additional analyses clarify the attribution of the reported gains and welcome any further questions or feedback during the discussion period.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response for 3 Reviewers

We thank the reviewers for their detailed feedback. TSADB learns a relational and temporal task representation for lightweight tool control. Our follow-up experiments separate information access from state organization and examine how the benefits depend on workflow scaffolding and policy training.

  1. Matched scaffolding and memory: Adding TSADB's recovery and stopping rules raises RL-ReAct to 75.4% BIRD EX, compared with TSADB's 82.0%; on 16+-call tasks, the scores are 44.7% and 60.3%. A flat encoder with the same cumulative schema, error memory, and ten-pair trace reaches 74.8%, leaving a 7.2-point overall gap. PPO over observable hand-built features reaches 75.6%.

  2. Alternative controllers: A frozen GPT-5 router receiving the structured information as text reaches 74.4% EX at 1,410 ms/step, compared with 82.0% at 846 ms/step for PPO-based TSADB. Four rounds of behavior cloning under the same 2,000-episode interaction budget reach 78.3%. Additional open-weight experiments compare decoupled control with end-to-end multi-turn RL.

  3. Evaluation and implementation: We have corrected three overall-accuracy entries in Table 4 (p. 8), which now separates the 800-task and full-development results. We also specify the relevance warmup, training-only reward information, recovery operations, tool masks, validation rules, and evaluation splits.

The revised manuscript contains these controls in Section 5.5 (pp. 8–9) and Appendices A–B and G. Our individual responses give the corresponding table and page locations.

AuthorsOpenReview ↗

Final Rebuttal Summary and Evidence Guide

We would like to express our heartfelt gratitude to the reviewers and the Area Chair for their careful consideration and constructive discussion.

To facilitate the subsequent assessment, we summarize the key controlled evidence and reporting clarifications in the latest revised manuscript.

1. State representation beyond memory and workflow scaffolding (ko6E; anQ5).

Under the same PPO procedure, reward, policy-head architecture, and full workflow scaffold, TSADB improves over memory-matched flat encoding by 7.2 percentage points overall on BIRD and 14.4 points on 16+-call tasks. The representation bridge further compares memory-matched flat encoding (74.8%), four-branch MLP-only encoding (77.4%), and full TSA (82.0%), distinguishing information access, input organization, and relational/temporal encoding along a controlled substitution sequence (Tables 5 and 24, pp. 9 and 35).

The State × Scaffold factorial shows that TSA retains an 8.6-point overall advantage and a 15.1-point long-task advantage even after jointly removing tool masking, adaptive recovery/stopping, and argument validation (Table 15, p. 24).

2. Alternative controller architectures and training strategies (anQ5; KG2v).

A structured-input GPT-5 text router achieves 74.4% BIRD EX, while four-round iterative behavior cloning reaches 78.3% under the same 2,000-rollout interaction budget, compared with 82.0% for PPO-based TSADB (Tables 5 and 13). These comparisons complement the within-framework representation controls rather than isolating architecture or training effects individually.

With the same Llama-3.1-8B backbone and 2,000-episode budget, decoupled TSADB reaches 66.9% EX at 72 GPU-hours, compared with 64.2% at 185 GPU-hours for the stronger evaluated end-to-end generative RL control (Table 22, p. 30).

3. Baseline fairness, evaluation integrity, and reporting (all reviewers).

The revised manuscript distinguishes training-free workflow comparisons from PPO-trained controls, specifies RL-baseline trainable modules and reward parameters, and documents relevance warmup, recovery, validation, and data partitions. We also corrected Table 4's mixed subset/full-development entries.

The expanded interface audit covers all seven training-free baseline families across BIRD-SQL, Spider 2.0-Lite, and DS-Bench (Table 25, p. 36). The strongest backbone-matched native-interface baseline scores are 73.2%, 35.3%, and 46.1%, respectively, versus shared-interface scores of 72.0%, 34.2%, and 45.2%.

Following the latest reviewer feedback, the abstract now reports these native-interface comparisons and the memory-matched representation gaps alongside the headline results.

Together, these additions provide complementary within-framework and system-level evidence for assessing the contribution of structured task-state learning under the stated experimental conditions. All results and protocols are incorporated into the revised manuscript.

We sincerely appreciate the reviewers' and the Area Chair's consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.