acceptodds
Under review as a conference paper at ICLR 2027

TACO: Reinforcing Multi-Turn Table Agents with Hierarchical Credit Assignment

Abstract

Table agents answer questions over multiple turns, where each turn writes code, executes it, and reads the result. Reinforcement learning on the final answer gives every decision the same credit. Credit assignment promises finer supervision, yet existing methods bring limited gains on table agents. We show that credit should be assigned hierarchically, first across turns and then within each turn, because a turn often chains several table operations of which only some are incorrect. Existing token-level methods, however, hardly distinguish the contributions of different operations within a turn. We further find that hindsight, the information revealed after a turn, shifts the probabilities of its tokens differently for correct and incorrect operations. We therefore propose TaCo, a hierarchical Table Credit Optimization framework for multi-turn table agents. Across turns, TaCo compares each turn with turns that received the same observations. Within a turn, a hindsight teacher, a periodically synchronized copy of the policy, rescores the turn with its execution feedback and a correct trajectory. The turn credit sets the direction of the update, and the probability gap between the teacher and the policy sets its size for each code and answer token. Extensive experiments on four table reasoning benchmarks across three model scales demonstrate that TaCo consistently improves multi-turn table agents and generalizes to benchmarks outside its training data.

Primary area:
foundation or frontier models, including LLMs
Keywords:
table reasoning, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oYLcOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies credit assignment for multi-turn table agents trained with GRPO. The proposed method, TACO, assigns credit in two stages. Across turns, it compares each turn with turns from other rollouts that saw the same observation history, as in GiGPO. Within a turn, a periodically synchronized copy of the policy re-scores the code and answer tokens with hindsight context: the turn's execution feedback and a correct sibling trajectory. The teacher–policy log-probability gap then scales each token's advantage within a bounded range, with the sign set by the turn credit. Experiments train Qwen2.5-Coder-Instruct at 3B, 7B and 14B on TableInstruct and WikiTQ. They evaluate on TableBench and WikiTQ (in-domain) and FinQA and HiTab (out-of-domain). TACO improves the four-benchmark average over GRPO by 2.0–2.5 points and over the best credit-assignment baseline by 0.9–1.7 points.

Critical strengths
  • Section 3 asks why existing credit methods underperform on code agents. The error taxonomy and the per-token credit comparison in Figs. 2–3 motivate the design directly.
  • TACO needs no value model and no extra rollouts. Its only overhead is one gradient-free teacher pass, and inference cost is unchanged.
  • Fig. 5 is useful as it keeps GiGPO's turn credit fixed and swaps only the source of the intra-turn signal. The hindsight-context study in Table 3 is also informative.
Critical weaknesses
  • Nothing in the method uses table structure. It relies on agents that write code, get deterministic execution and see informative printed output. Text-to-SQL agents, data-science notebook agents, math agents with a Python tool and coding agents all have that. So the "not designed for table agents" in the motivation doesn’t mark a real difference.

  • Appendix B.3 states that TACO without the intra-turn credit equals GiGPO. The only change is defining the anchor as the full observation history. The contribution list should credit this stage to GiGPO explicitly.

  • The intra-turn credit is incremental over self-distillation. SDPO (Hübotter et al., 2026) already conditions a self-teacher on runtime errors and a successful sibling's solution. The paper cites this, but mainly as motivation.

  • As reported in Section 5.4, the tasks considered in the experiments in this paper average up to 4 turns per task - this brings into question the effectiveness of turn-level credit assignment for short-horizon tasks and if the experiments demonstrate the effectiveness of the algorithm TACO proposed in this paper. The introduction cites ReasonTabQA and CompTab as motivation but never evaluates on them. Further lack of variance reporting makes it unclear whether a part of the gain is noise of experimentation.

  • Everything is on one model family (Qwen2.5-Coder) that is outdated.

Questions for authors
  • How does TACO compare with SDPO-style training in the same multi-turn setup? What does the bounded, sign-from-A_t weighting add?
  • What is the distribution of sibling-group sizes by turn index? What fraction of turns after turn 0 have a non-singleton group?
  • How do gains over GRPO and GiGPO vary with the number of turns and operations per turn?
Reviewer_RbiwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TACO for multi-turn, code-execution-based table question answering. The method first computes turn-level advantages through GiGPO-style grouping. It then uses a self-teacher conditioned on execution feedback and successful trajectories from the same group to recompute token probabilities for the current action, adjusting the update magnitudes of code and answer tokens with bounded weights. Experiments cover Qwen2.5-Coder models at 3B, 7B, and 14B scales and four table question-answering tasks. The results show that TACO improves average performance across all model scales, and component ablations support the contributions of both turn-level and within-turn signals.

Critical strengths
  • The experimental coverage is fairly comprehensive, with good control over the main training conditions.

  • The design is lightweight, and the ablations are targeted.

Critical weaknesses
  • The most closely related baselines are missing, so the benefits of the new design have not been isolated. SDPO already conditions a self-teacher on execution feedback or successful trajectories, provides training signals through token-level probability differences, and investigates its combination with GRPO. The turn-level component of this paper builds on GiGPO, yet the experiments do not compare against SDPO or GiGPO combined with SDPO under matched conditions. Removing within-turn credit assignment also removes the teacher signal, while GTPO/KTAE in Figure 5 do not have access to the same hindsight information. Therefore, the current experiments do not establish whether the weighting scheme in Equations (8)–(10) outperforms a direct combination of existing methods.

  • The labels in Figure 3 are insufficient to validate operation correctness. Appendix C.4 labels an operation as correct if it appears in another successful trajectory, and incorrect otherwise. This can misclassify alternative but equivalent correct implementations, while also failing to rule out redundant or incorrect operations within successful trajectories. Furthermore, Appendix C.5 first averages the two token categories separately within each turn. Consequently, Figure 3 does not directly demonstrate that the hindsight gap can localize errors, despite this being a central part of the paper's proposed mechanism.

Questions for authors
  • Under matched teacher, hindsight information, token masks, and training budgets, does TACO outperform SDPO and GiGPO combined with direct self-distillation?

  • Could the authors revalidate Figure 3 using independently human-annotated operation correctness labels? If the weights are shuffled within each turn or replaced with their within-turn averages, do the improvements persist?

Reviewer_SyCGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TACO, a hierarchical credit assignment framework for reinforcement learning of multi-turn table-reasoning agents. It combines GiGPO-style turn-level advantages with token-level reweighting derived from a hindsight-conditioned teacher–student log-probability gap, using execution feedback and successful peer trajectories. The resulting bounded, sign-preserving weights refine learning signals for code and answer tokens without requiring a separately trained critic or additional rollouts. Experiments on four table-reasoning benchmarks with 3B, 7B, and 14B models report improved average performance over GRPO and several credit assignment baselines, complemented by component ablations and comparisons of different hindsight contexts.

Critical strengths
  1. Relevant problem and practical approach. The paper addresses the mismatch between turn-level credit and the multiple operations within a table-processing code block. Using execution feedback to refine token weights without additional rollouts or a separately trained critic makes the approach practically appealing.

  2. Empirical support for the combined design. TACO achieves the highest four-benchmark average among the reported baselines at all three model scales. Removing either credit-assignment level reduces performance, supporting the practical value of combining turn-level and intra-turn signals.

Critical weaknesses
  1. Insufficient validation of the credit signal. Figure 3 uses operation occurrence in successful reference trajectories as a proxy for correctness. This can mislabel valid alternative operations, leaving the claimed ability of the hindsight gap to distinguish correct from incorrect operations insufficiently supported—a key gap in the paper’s justification for its credit refinement.

  2. Unclear robustness of the incremental gains. Although results are averaged across training seeds, the paper does not report seed counts or variability. Given average improvements of 0.9–1.7 points over the strongest reported baselines, this omission makes it difficult to assess the reliability of the gains that establish the method’s practical contribution.

  3. Limited methodological novelty. The turn-level estimator follows GiGPO, while hindsight scoring builds on prior self-distillation work. The contribution therefore rests largely on coupling these components through bounded token reweighting. Although the ablations demonstrate practical benefits, the analysis provides limited new insight into this coupling, making the methodological advance relatively incremental.

Questions for authors
  1. Operation correctness — clarification and targeted validation. How does Figure 3 handle an operation that is correct but absent from the successful reference trajectory? Could you independently check a small subset to establish whether the reported separation reflects actual operation correctness?

  2. Novelty — clarification. What is the specific methodological contribution beyond GiGPO’s turn-level advantages and the hindsight scoring used in prior self-distillation work? Which experiment most directly supports the benefit of your particular way of combining them?

  3. Robustness — clarification. How many training seeds were used, and what is the variation across seeds for TACO and the strongest baselines? Are the reported improvements consistent across individual runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.