- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on improving the turn-level weight assignment for multi-turn OPD. The authors view OPD from the perspective of policy gradients and show that the gradient decomposes into a calibrated distillation component and an orthogonal return-driven residual. The proposed method, TACT, uses conditionally centered signals to provide unbiased single-sample estimates from the batch, pools statistics across up to 13 discrete turn classes, and gates a residual RL step when the signal fails to explain reward variance. Evaluated across ALFWorld, ScienceWorld, WebShop, Search-QA, and -bench, TACT effectively outperforms various baselines.
Critical strengths
- The proof is sound and the resulting formulation is elegant.
- Solid experimental results comparing well-chosen baselines and showing consistent empirical gains
Critical weaknesses
- TACT needs to pool estimates into 13 coarse, hand-crafted buckets, which can be suboptimal and may require extra engineering effort; this manual design might also affect performance when applied to other tasks.
- Missing efficiency analysis. The number of turns and reasoning tokens can be essential for multi-turn reasoning tasks. Without these results, it is hard to judge fairly whether the method is really better than the baselines.
Questions for authors
- The derivation does not seem limited to multi-turn tasks. Is it possible to extend it to more general tasks?
- Instead of implicitly inferring the advantage, how about directly learning a value model? Can you discuss the potential trade-offs between the two paradigms?