- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
TIDE is a framework that aims to improve multi-turn LLM agent training by adaptively doing teacher-guided distillation with RL. It adaptively switches from teacher supervision to normal RL exploration, while using teacher–student disagreement and an estimate for action quality to decide when to shift. For Qwen models on WebShop and ALFWorld, results show consistent improvements over baselines.
Critical strengths
Thank you so much for submitting your paper. Overall, the concept is very clear and well explained. Below are the main strengths in no particular order.
- Clever idea and approach: I think the overall idea is easy but really clever and challenges a bit how current multi-turn agentic RL is done. Folks either do distillation or rely on env. rewards completely, but having something like a "teacher-inspired push start" is very interesting. Your framework then optimizes the adaptation between teacher influence across training stages. Your results also seem to show that this is effective.
- Practical integration: while the method looks quite fancy and complicated, I think a big strength is the practical integration to already existing OPD pipelines. There, you already have all the information needed for TIDE (as also for ATOD). This makes the approach a practical extension of an existing OPD–RL pipeline, and also one already supporting GiGPO-style grouping. Compute overhead is also negligible, further supporting the approach.
There are also other points that I appreciated:
- Ablations: principled and support the results.
- Presentation: Tables are clear, but I really enjoyed Fig. 1. Maybe you can increase the font size of the last row icon/symbol explanations.
- Reporting: overall very good, especially the efficiency reporting.
Critical weaknesses
Unfortunately, despite the mentioned strengths, there are some weaknesses that make your assessment flawed in some cases and raise further questions.
- Use of prior work results: including more baselines is definitely a good thing, but re-using results from prior work directly is something that I am not comfortable with. This is because you cannot make sure that their protocols (both for training and evaluation), and even their HW setup, would not change the results meaningfully. Doing this in Table 2 is a major flaw in my opinion and I cannot rely on your comparison for the methods therein. I digged a bit deeper and can tell you exactly why in two cases: 1) SDAR reports WebShop results on 128 validation tasks, whereas TIDE uses 500 evaluation tasks. 2) The original GiGPO setup also specifies 15 interaction steps and evaluation temperature 0.4, compared with 30 steps and temperature 1.0 for TIDE. This simple mismatch can, in some cases, result in a major difference. In particular, the comparison of 83.2% for TIDE against 82.8% for SDAR does not establish superiority under a shared protocol. Even if it fits your evaluation narrative here, it becomes difficult to say where the performance gaps come from. Is it TIDE's overall method, the configuration, the HW setup, etc.? Until now, as long as this is not verified under your setup and protocols, these comparisons are unfair.
- Advancement over hybrid methods: while I acknowlegde the good motivation behind TIDE, I think its advancement over closely related hybrid methods seems to be relatively small (if I am not mistaken). In particular, ATOD already combines an OPD-to-RL schedule with turn-level disagreement and uncertainty weighting. TIDE’s contribution therefore lies in the adaptive schedule and the addition of outcome information into the local weighting. So it is not particularly novel when it comes to the coordination of training stages and individual turns. Am I correct in this assessment? TIDE's adaptation also shows improvements but these are moderate over ATOD, so is TIDE an incremental extension of it?
Unfortunately, I cannot recommend a paper with imported results as I think this is not a good practice. You can happily change my mind in the rebuttal with updated results or the willingness to remove the imported ones.
There are some other (mostly weaker) points that I would like to rephrase as questions (down below) and are not critical.
Questions for authors
As mentioned above, there are some non-critical but nevertheless important points that I would like to phrase as questions to the authors.
1) Multiplicative versus Additive Weighting: Could the advantage of multiplicative fusion partly reflect a change in overall distillation strength? I briefly looked into App.A4 which defines the additive weight as ((q+d)/2), whereas TIDE uses (qd). Since both quantities are normalized to ([0,1]), the additive weight is always at least as large as the multiplicative one, correct? So this ablation changes both the allocation of supervision across turns and the effective balance between OPD and RL? Would the advantage also show if the variants were matched for average OPD weight or given comparable tuning of the distillation coefficient?
2) Repeated Contexts: How well does the full method transfer to environments where exact observation histories repeat less frequently? In Fig.6, you report high matched-history coverage after early training on WebShop/ALFWorld. But environments with more variable and/or noisy observations may provide fewer useful matches. Again, in App.A4, you show that unmatched turns receive a zero relative action value and retain disagreement-based distillation, but receive no value-driven RL update. How does this affect the intended transition toward RL when matching coverage remains low? Would a trajectory-level RL fallback be useful then?
Just some ideas, as I said, nothing critical.
Also, please reduce the use of AI-slop semicolons ";" in your sentences and paragraphs. This makes it really tough to read.
4 comments
Response to Question 1: Mean-matched multiplicative versus additive fusion
The original Additive Fusion uses rather than the unnormalized sum , keeping its range in to match the range of the product . Matching the range, however, does not match the average effective OPD strength: for , . We therefore add a mean-matched control to isolate the fusion rule from overall distillation strength.
To isolate the fusion rule, we add a fixed-scale, mean-calibrated additive control. For each benchmark, one token-weighted scaling constant is estimated from existing TIDE rollout logs and frozen before the new runs. It is never recomputed per turn, trajectory, batch, or update. This preserves additive allocation while matching its expected OPD scale to multiplicative TIDE; all other settings remain unchanged.
| Local OPD Fusion | OPD-Scale Calibration | WebShop Score | WebShop SR | ALFWorld Seen | ALFWorld Unseen |
|---|---|---|---|---|---|
| TIDE | Reference | ||||
| Additive Fusion | Range-Matched | ||||
| Additive Fusion (Mean-Matched) | Mean-Matched |
Results. After matching the additive variant's overall OPD scale, TIDE remains higher by 13.5 points in WebShop Score, 6.7 in SR, 14.3 on ALFWorld seen, and 15.9 on unseen success. These consistent gains show that multiplicative fusion improves turn-level allocation beyond merely reducing average distillation strength.
Response to Question 2: Low matched-history coverage and RL fallback
For a turn without a value-discriminative matched group, TIDE sets and . The turn retains disagreement-weighted OPD, while the global handoff continues to operate from batch-level disagreement. Figure 6 shows that valid-group coverage becomes high and stable on both evaluated environments after early exploration.
To test sensitivity to reduced coverage, we randomly drop 20% of the values only among turns that already have a valid, value-discriminative match. Dropped turns use the same fallback as naturally unmatched turns; all other settings remain unchanged.
| Method | Valid-value dropout | WebShop Score | WebShop SR | ALFWorld Seen | ALFWorld Unseen |
|---|---|---|---|---|---|
| TIDE | 0% | ||||
| TIDE + Value Dropout | 20% |
Results. Dropping 20% of otherwise valid values reduces WebShop Score, WebShop SR, ALFWorld seen, and unseen success by 2.0, 2.0, 2.4, and 1.8 points, respectively. The consistent decline demonstrates the importance of matched-history relative action value, while the moderate degradation shows that TIDE remains robust when some local values are unavailable. Because dropout removes the value signal from both OPD allocation and local RL credit, the experiment supports their joint contribution.
Response to Minor Suggestions: Presentation revisions
We will revise the prose to remove unnecessary semicolons, split overloaded sentences, and improve readability.
Thank you for the detailed responses!
My points are mostly addressed, however, I still struggle with my biggest concern, which is the import of prior work results.
You two controls show TIDE works, but what does that mean for the paper? Will you remove the imported results completely? Can you update the paper accordingly?
In my honest opinion, there should be no prior work results if you have not verified them yourself.
Thank you for your careful reading and constructive feedback. We have uploaded a revised version of the paper, and all revised or newly added content is marked in blue so that the changes are easy to locate. The main text remains within the 10-page limit, and additional details and analyses are provided in the appendix.
The main changes are:
- Fairness of comparison: We added protocol-matched GRPO and GiGPO controls and a GiGPO+OPD hybrid under the same teacher and training budget (Table 3, Appendix B.5), together with an external 128-task protocol check (Appendix B.8). TIDE retains its advantage in these settings.
- Handoff schedule: We added pilot-matched fixed-window runs and cross-task transfer experiments (Table 3(a), Appendix B.6). The realized handoff trajectories are now shown in the main text as Figure 4, and the SearchQA evaluation is reported in Appendix B.1.
- Local allocation: We added a mean-matched additive fusion control and a value-coverage analysis (Table 3(b), Appendix B.7), showing that the improvement is not explained by the overall distillation strength alone.
- Process-quality analysis: We added an all-turn AUROC/Spearman analysis showing that teacher–student disagreement alone is only weakly associated with externally judged process quality (Section 2).
- Related work: We clarified the relationship to ATOD and RetireOPD, including methodological differences and an aggregate comparison (Appendix B.9).
- Language: We revised the prose to remove unnecessary semicolons and improve readability.
We have tried to address each concern as directly as we could, and we are aware that the revision may still fall short in places. If you feel that the main issues have been addressed, we would be sincerely grateful if you would reconsider your assessment of the paper. If any concern remains, we would be very glad to discuss it further.
We sincerely thank the reviewer for the careful protocol analysis, the thoughtful comparison with ATOD, and the constructive questions about our local weighting and matched-history assumptions. We address each concern separately below.
Response to Weakness 1: Imported prior-work results and protocol-matched reevaluation
We provide two complementary controls. First, we reproduce complete GiGPO under the TIDE protocol, changing the learning method while holding the training and evaluation settings fixed:
Results. Under this shared protocol, TIDE exceeds GiGPO by 9.4 points in WebShop SR, 5.7 in WebShop Score, 2.7 on ALFWorld valid-seen, and 4.0 on valid-unseen, demonstrating TIDE's advantage under fully matched settings.
Second, we evaluate TIDE-1.5B and TIDE-7B under the original GiGPO/SDAR 128-task protocol on both benchmarks. The table reports means for consistency with the original studies.
Results. TIDE is stable across the two evaluation protocols, with every mean changing by at most 1.2 points. Under the external protocol, TIDE-1.5B improves over GiGPO by 5.6 points in WebShop Score and 13.1 in SR. At 7B, TIDE exceeds GiGPO on all three metrics and exceeds SDAR by 6.3 points on ALFWorld and 2.4 on WebShop Score, while remaining competitive on SR. Together with the matched GiGPO rerun above, these results show that TIDE's gains persist under both evaluation protocols.
Response to Weakness 2: Advancement beyond ATOD
As discussed above, our ATOD reproduction follows the official paper and code and includes complete T-DUR rather than only linear annealing. The methods differ at both the global and local levels:
With all remaining settings matched, complete ATOD and TIDE obtain:
TIDE is higher than complete ATOD on all three metrics at all three model scales, with gains of 2.4–2.9 points on ALFWorld Avg., 3.0–4.3 on WebShop Score, and 4.0–4.9 on WebShop SR. The pilot-matched fixed schedule further isolates the global distinction: TIDE remains higher by 2.1, 2.6, 0.8, and 2.3 points on WebShop SR, Score, ALFWorld seen, and unseen, respectively. The GiGPO–OPD controls isolate a simple combination of process-level RL and teacher supervision, over which TIDE remains higher by 6.6, 4.7, 1.2, and 1.5 points on the same four metrics. These two controls correspond to the global-timing and local-coordination distinctions above, showing that TIDE's advancement over ATOD is not limited to replacing the linear schedule or merely adding outcome information to a local weight.