acceptodds
Under review as a conference paper at ICLR 2027

TIDE: Teacher-Student Transition via Informative Distillation and Exploration for Agentic RL

Abstract

Effective multi-turn agents require interaction strategies that coordinate information gathering, actions, and feedback over long horizons. GRPO is a reinforcement learning algorithm used to train these agents, but sparse trajectory-level rewards limit early exploration in small models. Recent methods augment RL with on-policy distillation (OPD) from a stronger teacher. However, a fixed mixture assumes that teacher guidance and reward optimization should retain a constant relative role throughout training and across interaction turns. This assumption can fail at two scales. Globally, as training progresses, maintaining strong distillation pressure can constrain the model from moving beyond the teacher's capabilities. Locally, teacher–student disagreement identifies where the student departs from the teacher, but cannot tell whether that departure is exploration supported by better outcomes or low-quality policy drift. Our methodological insight is that teacher guidance and reward optimization should be dynamically rebalanced over training and jointly allocated across turns. We instantiate this insight in \tide. Globally, \tide uses the measured disagreement trend as a practical schedule signal, advancing an OPD-to-RL handoff when discrepancy reduction becomes slow but remains positive and progressively increasing the relative weight of RL. Locally, \tide jointly modulates teacher-guided and reward-driven updates: relative action value and disagreement prioritize the OPD signal, whereas relative action value supplies the RL advantage and normalized disagreement reweights it across turns. Coupled with the global handoff, \tide allocates stronger teacher guidance early and gives reward-driven updates greater relative weight later in training. Experiments across multiple benchmarks, student scales, and controlled ablations support the effectiveness of TIDE's adaptive OPD–RL coordination.

Primary area:
reinforcement learning
Keywords:
Agentic RL, OPD

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_2oVYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

TIDE is a framework that aims to improve multi-turn LLM agent training by adaptively doing teacher-guided distillation with RL. It adaptively switches from teacher supervision to normal RL exploration, while using teacher–student disagreement and an estimate for action quality to decide when to shift. For Qwen models on WebShop and ALFWorld, results show consistent improvements over baselines.

Critical strengths

Thank you so much for submitting your paper. Overall, the concept is very clear and well explained. Below are the main strengths in no particular order.

  • Clever idea and approach: I think the overall idea is easy but really clever and challenges a bit how current multi-turn agentic RL is done. Folks either do distillation or rely on env. rewards completely, but having something like a "teacher-inspired push start" is very interesting. Your framework then optimizes the adaptation between teacher influence across training stages. Your results also seem to show that this is effective.
  • Practical integration: while the method looks quite fancy and complicated, I think a big strength is the practical integration to already existing OPD pipelines. There, you already have all the information needed for TIDE (as also for ATOD). This makes the approach a practical extension of an existing OPD–RL pipeline, and also one already supporting GiGPO-style grouping. Compute overhead is also negligible, further supporting the approach.

There are also other points that I appreciated:

  • Ablations: principled and support the results.
  • Presentation: Tables are clear, but I really enjoyed Fig. 1. Maybe you can increase the font size of the last row icon/symbol explanations.
  • Reporting: overall very good, especially the efficiency reporting.
Critical weaknesses

Unfortunately, despite the mentioned strengths, there are some weaknesses that make your assessment flawed in some cases and raise further questions.

  • Use of prior work results: including more baselines is definitely a good thing, but re-using results from prior work directly is something that I am not comfortable with. This is because you cannot make sure that their protocols (both for training and evaluation), and even their HW setup, would not change the results meaningfully. Doing this in Table 2 is a major flaw in my opinion and I cannot rely on your comparison for the methods therein. I digged a bit deeper and can tell you exactly why in two cases: 1) SDAR reports WebShop results on 128 validation tasks, whereas TIDE uses 500 evaluation tasks. 2) The original GiGPO setup also specifies 15 interaction steps and evaluation temperature 0.4, compared with 30 steps and temperature 1.0 for TIDE. This simple mismatch can, in some cases, result in a major difference. In particular, the comparison of 83.2% for TIDE against 82.8% for SDAR does not establish superiority under a shared protocol. Even if it fits your evaluation narrative here, it becomes difficult to say where the performance gaps come from. Is it TIDE's overall method, the configuration, the HW setup, etc.? Until now, as long as this is not verified under your setup and protocols, these comparisons are unfair.
  • Advancement over hybrid methods: while I acknowlegde the good motivation behind TIDE, I think its advancement over closely related hybrid methods seems to be relatively small (if I am not mistaken). In particular, ATOD already combines an OPD-to-RL schedule with turn-level disagreement and uncertainty weighting. TIDE’s contribution therefore lies in the adaptive schedule and the addition of outcome information into the local weighting. So it is not particularly novel when it comes to the coordination of training stages and individual turns. Am I correct in this assessment? TIDE's adaptation also shows improvements but these are moderate over ATOD, so is TIDE an incremental extension of it?

Unfortunately, I cannot recommend a paper with imported results as I think this is not a good practice. You can happily change my mind in the rebuttal with updated results or the willingness to remove the imported ones.

There are some other (mostly weaker) points that I would like to rephrase as questions (down below) and are not critical.

Questions for authors

As mentioned above, there are some non-critical but nevertheless important points that I would like to phrase as questions to the authors.

1) Multiplicative versus Additive Weighting: Could the advantage of multiplicative fusion partly reflect a change in overall distillation strength? I briefly looked into App.A4 which defines the additive weight as ((q+d)/2), whereas TIDE uses (qd). Since both quantities are normalized to ([0,1]), the additive weight is always at least as large as the multiplicative one, correct? So this ablation changes both the allocation of supervision across turns and the effective balance between OPD and RL? Would the advantage also show if the variants were matched for average OPD weight or given comparable tuning of the distillation coefficient?

2) Repeated Contexts: How well does the full method transfer to environments where exact observation histories repeat less frequently? In Fig.6, you report high matched-history coverage after early training on WebShop/ALFWorld. But environments with more variable and/or noisy observations may provide fewer useful matches. Again, in App.A4, you show that unmatched turns receive a zero relative action value and retain disagreement-based distillation, but receive no value-driven RL update. How does this affect the intended transition toward RL when matching coverage remains low? Would a trajectory-level RL fallback be useful then?

Just some ideas, as I said, nothing critical.

Also, please reduce the use of AI-slop semicolons ";" in your sentences and paragraphs. This makes it really tough to read.

4 comments

We sincerely thank the reviewer for the careful protocol analysis, the thoughtful comparison with ATOD, and the constructive questions about our local weighting and matched-history assumptions. We address each concern separately below.

Response to Weakness 1: Imported prior-work results and protocol-matched reevaluation

We provide two complementary controls. First, we reproduce complete GiGPO under the TIDE protocol, changing the learning method while holding the training and evaluation settings fixed:

Method under the TIDE protocolWebShop SR ↑\uparrowWebShop Score ↑\uparrowALFWorld Seen ↑\uparrowALFWorld Unseen ↑\uparrow
GRPO62.8±2.062.8\pm2.082.4±1.282.4\pm1.258.8±0.858.8\pm0.847.8±2.647.8\pm2.6
Matched GiGPO67.8±1.967.8\pm1.984.1±1.484.1\pm1.483.3±1.183.3\pm1.181.1±1.681.1\pm1.6
TIDE77.2±1.8\mathbf{77.2\pm1.8}89.8±1.2\mathbf{89.8\pm1.2}86.0±0.8\mathbf{86.0\pm0.8}85.1±1.5\mathbf{85.1\pm1.5}

Results. Under this shared protocol, TIDE exceeds GiGPO by 9.4 points in WebShop SR, 5.7 in WebShop Score, 2.7 on ALFWorld valid-seen, and 4.0 on valid-unseen, demonstrating TIDE's advantage under fully matched settings.

Second, we evaluate TIDE-1.5B and TIDE-7B under the original GiGPO/SDAR 128-task protocol on both benchmarks. The table reports means for consistency with the original studies.

ScaleMethodALFWorld Avg. ↑\uparrowWebShop Score ↑\uparrowWebShop SR ↑\uparrow
1.5BGiGPO86.786.783.183.165.065.0
1.5BTIDE85.285.288.788.778.178.1
7BGiGPO90.890.884.484.472.872.8
7BSDAR85.985.989.489.482.882.8
7BTIDE92.292.291.891.882.082.0

Results. TIDE is stable across the two evaluation protocols, with every mean changing by at most 1.2 points. Under the external protocol, TIDE-1.5B improves over GiGPO by 5.6 points in WebShop Score and 13.1 in SR. At 7B, TIDE exceeds GiGPO on all three metrics and exceeds SDAR by 6.3 points on ALFWorld and 2.4 on WebShop Score, while remaining competitive on SR. Together with the matched GiGPO rerun above, these results show that TIDE's gains persist under both evaluation protocols.

Response to Weakness 2: Advancement beyond ATOD

As discussed above, our ATOD reproduction follows the official paper and code and includes complete T-DUR rather than only linear annealing. The methods differ at both the global and local levels:

AspectATODTIDE
Global OPD–RL balancePredefined linear annealing scheduleAdvances the handoff online from the disagreement-reduction trend
Local OPD weightingReweights OPD through T-DUR using disagreement and student uncertaintyReweights OPD jointly using outcome-derived relative value and disagreement
RL creditTrajectory-level GRPOSigned turn-level relative action value
Scope of local coordinationModulates OPD allocationCoordinates the turn-level influence of both OPD and RL

With all remaining settings matched, complete ATOD and TIDE obtain:

ScaleMethodALFWorld Avg.WebShop ScoreWebShop SR
1.5BATOD83.683.685.585.573.273.2
TIDE86.086.089.889.877.277.2
3BATOD86.486.486.686.674.174.1
TIDE89.389.390.290.279.079.0
7BATOD89.389.389.189.179.079.0
TIDE92.192.192.192.183.283.2

TIDE is higher than complete ATOD on all three metrics at all three model scales, with gains of 2.4–2.9 points on ALFWorld Avg., 3.0–4.3 on WebShop Score, and 4.0–4.9 on WebShop SR. The pilot-matched fixed schedule further isolates the global distinction: TIDE remains higher by 2.1, 2.6, 0.8, and 2.3 points on WebShop SR, Score, ALFWorld seen, and unseen, respectively. The GiGPO–OPD controls isolate a simple combination of process-level RL and teacher supervision, over which TIDE remains higher by 6.6, 4.7, 1.2, and 1.5 points on the same four metrics. These two controls correspond to the global-timing and local-coordination distinctions above, showing that TIDE's advancement over ATOD is not limited to replacing the linear schedule or merely adding outcome information to a local weight.

Response to Question 1: Mean-matched multiplicative versus additive fusion

The original Additive Fusion uses (qt+dt)/2(q_t+d_t)/2 rather than the unnormalized sum qt+dtq_t+d_t, keeping its range in [0,1][0,1] to match the range of the product qtdtq_td_t. Matching the range, however, does not match the average effective OPD strength: for qt,dt∈[0,1]q_t,d_t\in[0,1], (qt+dt)/2≥qtdt(q_t+d_t)/2\ge q_td_t. We therefore add a mean-matched control to isolate the fusion rule from overall distillation strength.

To isolate the fusion rule, we add a fixed-scale, mean-calibrated additive control. For each benchmark, one token-weighted scaling constant is estimated from existing TIDE rollout logs and frozen before the new runs. It is never recomputed per turn, trajectory, batch, or update. This preserves additive allocation while matching its expected OPD scale to multiplicative TIDE; all other settings remain unchanged.

Local OPD FusionOPD-Scale CalibrationWebShop ScoreWebShop SRALFWorld SeenALFWorld Unseen
TIDEReference89.8±1.2\mathbf{89.8\pm1.2}77.2±1.8\mathbf{77.2\pm1.8}86.0±0.8\mathbf{86.0\pm0.8}85.1±1.5\mathbf{85.1\pm1.5}
Additive FusionRange-Matched74.7±2.774.7\pm2.767.4±3.867.4\pm3.870.7±2.570.7\pm2.570.1±2.270.1\pm2.2
Additive Fusion (Mean-Matched)Mean-Matched76.3±1.876.3\pm1.870.5±2.570.5\pm2.571.7±2.571.7\pm2.569.2±2.369.2\pm2.3

Results. After matching the additive variant's overall OPD scale, TIDE remains higher by 13.5 points in WebShop Score, 6.7 in SR, 14.3 on ALFWorld seen, and 15.9 on unseen success. These consistent gains show that multiplicative fusion improves turn-level allocation beyond merely reducing average distillation strength.

Response to Question 2: Low matched-history coverage and RL fallback

For a turn without a value-discriminative matched group, TIDE sets Qt=0Q_t=0 and Q~t=1\widetilde Q_t=1. The turn retains disagreement-weighted OPD, while the global handoff continues to operate from batch-level disagreement. Figure 6 shows that valid-group coverage becomes high and stable on both evaluated environments after early exploration.

To test sensitivity to reduced coverage, we randomly drop 20% of the QtQ_t values only among turns that already have a valid, value-discriminative match. Dropped turns use the same fallback as naturally unmatched turns; all other settings remain unchanged.

MethodValid-value dropoutWebShop ScoreWebShop SRALFWorld SeenALFWorld Unseen
TIDE0%89.8±1.2\mathbf{89.8\pm1.2}77.2±1.8\mathbf{77.2\pm1.8}86.0±0.8\mathbf{86.0\pm0.8}85.1±1.5\mathbf{85.1\pm1.5}
TIDE + Value Dropout20%87.8±1.287.8\pm1.275.2±1.875.2\pm1.883.6±0.783.6\pm0.783.3±1.683.3\pm1.6

Results. Dropping 20% of otherwise valid values reduces WebShop Score, WebShop SR, ALFWorld seen, and unseen success by 2.0, 2.0, 2.4, and 1.8 points, respectively. The consistent decline demonstrates the importance of matched-history relative action value, while the moderate degradation shows that TIDE remains robust when some local values are unavailable. Because dropout removes the value signal from both OPD allocation and local RL credit, the experiment supports their joint contribution.

Response to Minor Suggestions: Presentation revisions

We will revise the prose to remove unnecessary semicolons, split overloaded sentences, and improve readability.

Thank you for the detailed responses!

My points are mostly addressed, however, I still struggle with my biggest concern, which is the import of prior work results.

You two controls show TIDE works, but what does that mean for the paper? Will you remove the imported results completely? Can you update the paper accordingly?

In my honest opinion, there should be no prior work results if you have not verified them yourself.

Thank you for your careful reading and constructive feedback. We have uploaded a revised version of the paper, and all revised or newly added content is marked in blue so that the changes are easy to locate. The main text remains within the 10-page limit, and additional details and analyses are provided in the appendix.

The main changes are:

  • Fairness of comparison: We added protocol-matched GRPO and GiGPO controls and a GiGPO+OPD hybrid under the same teacher and training budget (Table 3, Appendix B.5), together with an external 128-task protocol check (Appendix B.8). TIDE retains its advantage in these settings.
  • Handoff schedule: We added pilot-matched fixed-window runs and cross-task transfer experiments (Table 3(a), Appendix B.6). The realized handoff trajectories are now shown in the main text as Figure 4, and the SearchQA evaluation is reported in Appendix B.1.
  • Local allocation: We added a mean-matched additive fusion control and a value-coverage analysis (Table 3(b), Appendix B.7), showing that the improvement is not explained by the overall distillation strength alone.
  • Process-quality analysis: We added an all-turn AUROC/Spearman analysis showing that teacher–student disagreement alone is only weakly associated with externally judged process quality (Section 2).
  • Related work: We clarified the relationship to ATOD and RetireOPD, including methodological differences and an aggregate comparison (Appendix B.9).
  • Language: We revised the prose to remove unnecessary semicolons and improve readability.

We have tried to address each concern as directly as we could, and we are aware that the revision may still fall short in places. If you feel that the main issues have been addressed, we would be sincerely grateful if you would reconsider your assessment of the paper. If any concern remains, we would be very glad to discuss it further.

Reviewer_Ah25OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TIDE, a two-scale framework for coordinating OPD and RL in multi-turn LLM agents. The method introduces a disagreement-triggered global handoff mechanism to dynamically adjust the balance between teacher guidance and reward optimization during training. It also incorporates turn-level modulation based on relative action values and teacher–student disagreement to allocate learning signals across interaction steps. Experiments on WebShop and ALFWorld across multiple student model scales demonstrate the effectiveness of the proposed approach.

Critical strengths
  1. The paper addresses an important challenge in agentic reinforcement learning: how to balance teacher supervision and autonomous exploration throughout training. The proposed two-scale perspective, considering both training-stage transitions and turn-level signal allocation, provides a meaningful framework for coordinating OPD and RL.
  2. The paper evaluates TIDE across multiple model scales and compares it with representative RL, distillation, and hybrid methods. In particular, the controlled comparisons with ATOD and the ablation studies on global scheduling and local modulation provide useful evidence supporting the proposed design.
Critical weaknesses
  1. In Section 2.3, the local OPD weight increases with both relative action value and teacher–student disagreement. However, when a high-value student action is assigned a negative teacher-relative log-probability difference, the OPD and RL updates may encourage opposing directions. It remains unclear how TIDE handles such conflicts and whether stronger distillation could suppress beneficial exploration. Further analysis of the interaction between these two learning signals would strengthen the methodological justification.
  2. Although the paper provides controlled comparisons with ATOD and several scheduling strategies, other recent approaches, such as RetireOPD, also adapt teacher guidance based on training dynamics and teacher–student disagreement. A more explicit comparison or discussion of the differences between these mechanisms would help establish the specific advantages of TIDE's global handoff strategy.
  3. The full TIDE framework is primarily evaluated on WebShop and ALFWorld. Although Appendix B.1 includes additional SearchQA experiments, these evaluate only the global handoff mechanism without local modulation. Consequently, the generalizability of the complete two-scale framework to other interactive environments and longer-horizon agent tasks remains insufficiently established.
Questions for authors
  1. How does TIDE handle situations where high-value student actions disagree with the teacher and the OPD and RL signals potentially conflict? Could the authors provide additional analysis of such cases?
  2. Could the authors clarify the key differences between TIDE's disagreement-triggered handoff and other adaptive teacher-guidance approaches, particularly RetireOPD? Additionally, how does the controller behave when teacher–student disagreement stops decreasing or begins to increase?
  3. Could the authors discuss whether the complete framework, including both global and local mechanisms, is expected to generalize beyond WebShop and ALFWorld?
3 comments

We thank the reviewer for the thoughtful questions about learning-signal coordination, adaptive teacher guidance, and the scope of the complete framework. We address each concern separately.

Response to Weakness 1 and Question 1: Global–local coordination when OPD and RL point in different directions

In TIDE, OPD and RL updates are combined within the same token-level advantage. Let qtq_t and dtd_t denote the normalized relative value and disagreement:

At,jTIDE=dt(βkQt+λkqtδt,j).A^{\mathrm{TIDE}}_{t,j} =d_t\left(\beta_k Q_t+\lambda_k q_t\delta_{t,j}\right).

When a high-value action has Qt>0Q_t>0 but its teacher-relative log-probability difference satisfies δt,j<0\delta_{t,j}<0, OPD and RL indeed point in opposite directions. However, TIDE does not resolve this conflict with fixed weights; the global handoff changes their priority across training stages. OPD dominates early, using dense teacher guidance to stabilize learning. As the handoff progresses, RL gradually becomes dominant, preserving teacher-deviating actions supported by outcome evidence. This prevents teacher guidance from persistently suppressing beneficial exploration and creates room for the student to improve beyond the teacher policy. A nonzero residual OPD term retains teacher guidance in the later stage. Local modulation further selects turns through the product qtdtq_td_t: a turn receives high weight only when both relative value and disagreement are high, while a low value in either signal suppresses the product. This conjunctive selection motivates multiplicative rather than additive fusion.

MethodWebShop SR ↑\uparrowWebShop Score ↑\uparrowALFWorld Seen ↑\uparrowALFWorld Unseen ↑\uparrow
Cosine Handoff73.2±1.473.2\pm1.487.3±0.787.3\pm0.785.0±1.485.0\pm1.481.3±1.581.3\pm1.5
w/o Process Reward72.2±2.672.2\pm2.684.4±1.784.4\pm1.780.0±2.180.0\pm2.179.1±2.279.1\pm2.2
w/o Disagreement74.4±1.674.4\pm1.685.0±1.085.0\pm1.083.6±1.483.6\pm1.479.9±1.579.9\pm1.5
Additive Fusion67.4±3.867.4\pm3.874.7±2.774.7\pm2.770.7±2.570.7\pm2.570.1±2.270.1\pm2.2
TIDE77.2±1.8\mathbf{77.2\pm1.8}89.8±1.2\mathbf{89.8\pm1.2}86.0±0.8\mathbf{86.0\pm0.8}85.1±1.5\mathbf{85.1\pm1.5}

Globally, TIDE improves WebShop SR over Cosine Handoff by 4.0 points. Locally, removing process reward or disagreement reduces WebShop SR by 5.0 and 2.8 points, respectively; replacing the product with Additive Fusion reduces it by 9.8 points. The other three metrics show the same overall pattern, supporting disagreement-based global handoff, joint value/disagreement selection, and multiplicative fusion.

Response to Weakness 2 and Question 2: Relationship to RetireOPD and controller behavior

As discussed above, RetireOPD is closely related concurrent work: its first arXiv version appeared on September 17, 2026, immediately before the ICLR 2027 abstract deadline, and its revised version appeared after the full-paper deadline. The methods differ at two levels. Globally, RetireOPD removes teacher supervision after disagreement stagnates and the student reaches a specified fraction of teacher performance. TIDE does not require a teacher-relative performance threshold; it performs a gradual, monotone handoff based on the disagreement-reduction trend and retains residual OPD. Locally, RetireOPD adapts global teacher retirement, whereas TIDE additionally uses outcome-derived relative action value and disagreement to modulate both OPD and RL across turns.

The results reported by the two papers are:

ModelMethodALFWorld SuccessWebShop ScoreWebShop SR/Acc
Qwen2.5-1.5BTIDE86.089.877.2
RetireOPD89.886.675.8
Qwen2.5-3BTIDE89.390.279.0
RetireOPD93.885.877.3
Qwen2.5-7BTIDE92.192.183.2
RetireOPD95.391.784.4

Among the reported results, TIDE has higher WebShop Score at all three model scales and higher WebShop success at 1.5B and 3B, showing consistent competitiveness on WebShop.

When disagreement stops decreasing or begins to increase, the controller behavior follows Eqs. (4)–(5):

Observed disagreement trendConditionController action
Still decreasing rapidlyIk>ζI_k>\zetaHold the current handoff state; teacher alignment is still improving substantially
Decreasing slowly but positively0<Ik≤ζ0<I_k\le\zetaIncrease the handoff state by η\eta, decreasing OPD and increasing RL
Temporarily flat or increasingIk≤0I_k\le0Hold the current state without reversing prior handoff progress

When disagreement becomes flat or increases, TIDE holds rather than reverses the current handoff state; if slow positive reduction resumes, the handoff continues from that state. Figure 7 shows the realized handoff trajectories produced by this controller on WebShop, ALFWorld, and SearchQA. We further fix TIDE's realized transition window as a pilot-matched schedule and transfer that fixed window across tasks:

MethodWebShop SRWebShop ScoreALFWorld SeenALFWorld Unseen
TIDE77.2±1.877.2\pm1.889.8±1.289.8\pm1.286.0±0.886.0\pm0.885.1±1.585.1\pm1.5
Pilot-Matched Fixed Schedule75.1±1.575.1\pm1.587.2±0.887.2\pm0.885.2±2.285.2\pm2.282.8±2.082.8\pm2.0
Fixed Schedule Transfer (ALFWorld →\rightarrow WebShop)74.9±1.574.9\pm1.586.9±1.186.9\pm1.1——
Fixed Schedule Transfer (WebShop →\rightarrow ALFWorld)——84.8±1.884.8\pm1.883.1±1.983.1\pm1.9

TIDE exceeds the task-matched pilot-fixed schedule by 2.1, 2.6, 0.8, and 2.3 points on the four metrics, respectively; every transferred fixed-window result also remains below TIDE. These results show that reproducing the schedule shape with a fixed window does not recover the adaptive controller's performance and demonstrate the practical value of avoiding manually specified handoff timing for each task.

Response to Weakness 3 and Question 3: Generalization of the complete two-scale framework

We evaluate complete TIDE on WebShop and ALFWorld, covering web navigation and embodied textual control at three student scales. SearchQA additionally tests the global handoff across seven retrieval-augmented QA datasets:

EvaluationTask coverageTIDE components evaluatedResult and evidential scope
WebShop and ALFWorldWeb navigation and embodied textual control; 1.5B, 3B, and 7B studentsComplete global + local frameworkDirect evidence for full TIDE on the two standard interactive benchmarks
SearchQANQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, MuSiQue, and BamboogleGlobal handoff only39.4%39.4\% macro-average EM versus 36.3%36.3\% for OPD; higher on six of seven subsets

Results. The complete framework is consistently validated across two distinct interactive environments and three model scales, while the global controller improves macro-average EM from 36.3%36.3\% to 39.4%39.4\% on SearchQA and performs better on six of seven subsets. Because the local mechanism uses terminal rewards and matched histories to estimate relative action value, these results support its application to multi-turn tasks that provide both signals.

Thank you for clarifying the differences from RetireOPD and adding the scheduling comparisons. The aggregate ablations support TIDE’s design, but my main concern remains how it handles high-value student actions when the OPD and RL signals conflict. The SearchQA results also evaluate only the global handoff, leaving the generalization of the full framework beyond WebShop and ALFWorld open. I therefore maintain my original score of 2.

Reviewer_Y6kPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Multi-turn LLM agents trained with RL get only a sparse success signal at the end of each episode, so small models learn slowly. Distilling from a stronger teacher on the student's own rollouts (on-policy distillation, OPD) gives a dense signal, but existing hybrids mix the two with a fixed or preset weight that is the same for every turn.

TIDE changes the mix at two levels. Over training, it tracks how fast teacher-student disagreement is falling, and once the decline slows it shifts weight step by step from distillation to RL. Within each episode, it weights each turn by how much the student disagrees with the teacher and by a GiGPO-style estimate of whether that turn's action led to a better outcome than alternatives taken from the same state. The teacher is a 7B model trained with GRPO on the same task.

On WebShop and ALFWorld with Qwen2.5 students from 1.5B to 7B, TIDE beats a reproduced hybrid baseline (ATOD) that uses the same teacher and budget, on both benchmarks at every scale. Ablations at two scales show every alternative schedule and every removed component doing worse, and a retrieval QA experiment tests the schedule alone.

Critical strengths

S1. Consistent gains over a matched hybrid baseline.

The authors reproduce ATOD with the same teacher, environment, budget and evaluation, and TIDE beats it on every averaged metric at every scale: WebShop success by 4.0, 4.9 and 4.2 points at 1.5B, 3B and 7B, and ALFWorld by 2.4, 2.9 and 2.8. Even with three seeds the ALFWorld gaps are clearly significant (Welch p <= 0.02), and the WebShop gaps mostly are (p about 0.02-0.03 at 3B and 7B, 0.09 at 1.5B). The 1.5B student also ends above its own teacher (86.0 vs 80.5 on ALFWorld), and the teacher's extra computation costs only 4.8-11.1% of update time (Figure 5a). This is the paper's real result.

S2. A complete and honestly scoped description.

Every normalization, fallback and ablation is defined exactly (Appendix A.4), the realized handoff is plotted rather than asserted (Figure 7), the ablations are repeated at 3B (Table 6), and the authors are careful about what their signals mean (disagreement is "not a direct measurement of teacher utility"). This makes the concerns below checkable and fixable.

Critical weaknesses

W1. The paper does not show which part of TIDE produces the gain.

At 1.5B, changing any single component, the schedule or any turn weight, drops WebShop success to 67.4-74.4, around ATOD's 73.2; only the exact full configuration reaches 77.2. No validation split is described, so the configuration may have been chosen on the test sets. Three simpler explanations remain open. First, TIDE's RL part uses GiGPO's turn-level credit, not GRPO's, yet GiGPO is never run under the same setup, and its published 1.5B ALFWorld score (86.7) already exceeds TIDE's 86.0. Second, the realized handoff (Figure 7) is a delayed linear ramp: no shift until about step 51 on WebShop, then a steady climb to full RL over about 60-70 updates. The linear and cosine baselines start shifting at step 1, so none tests this shape, and on ALFWorld seen the adaptive trigger beats the linear schedule by only 0.3. Third, the turn weights reduce the effective amount of distillation, and Figure 3's early lead over OPD appears before the handoff has started.

W2. The statistics are too thin for the component claims, and some reported numbers cannot be right.

Everything uses three seeds, and several "necessary component" claims rest on gaps of 1-2 points (1.9 and 2.0 at 3B, p about 0.18). The OPD rows' ALFWorld standard deviations cannot come from three seeds on the 140-task split. With whole-task counts only certain values are possible (for Clean, with 27 tasks: 0, 2.1, 3.7, 4.3, ...), yet OPD reports 4.0, 3.3 and 3.0; only 3 of its 21 cells are possible at all. The rows copied from prior work were not measured on this split either, so they are not directly comparable. Finally, Table 1's motivating diagnostic uses a judge described as blind to outcomes, yet its lowest- and highest-quality groups have exactly 0.0% and 100.0% success, which suggests ties were broken using the outcome.

Questions for authors

Q1. Matched GiGPO baselines. Please run GiGPO with your turn-value settings, with and without its episode-level term, and GiGPO plus OPD with a fixed 1:1 and a linear schedule but without TIDE's turn weights, at 1.5B with 3 seeds. Which RL advantage does ATOD use?

Q2. Schedule shape and tuning. Does a preset schedule that copies Figure 7 (no shift until step 51, then linear to full RL by step 114) match TIDE on WebShop, and does a start step tuned on WebShop transfer to ALFWorld? On which data were the schedule and turn-value hyperparameters chosen, and which student checkpoint is reported?

Q3. Standard deviations. How were the OPD standard deviations in Table 2 computed? Please release per-seed success counts per task type for every row you ran in Tables 2, 3 and 6. Relatedly, ATOD's per-type spreads are identical at all three scales and equal to those of TIDE at 3B.

Q4. Table 1. How were ties broken when taking the bottom and top 20% of a 1-3 integer score? Please also report how well disagreement predicts the judge score over all turns (for example, AUROC).

Minor suggestions (do not affect the score).

(a) State Figure 2's student scale, and reconcile its OPD disagreement (about 0.22 falling to 0.13, consistent with Table 1's 0.131) with Figure 3(c)'s (about 0.32 falling to 0.24).

(b) Fix the empty "Normalization pseudocode." heading on page 16 and the judge prompt's JSON on page 14, which prints LaTeX double backticks. The OpenReview abstract also shows an unexpanded LaTeX macro instead of the method name.

3 comments

We thank the reviewer for the detailed questions about component attribution, experimental protocol, and statistical reporting. We address each issue separately below.

Response to Weakness 1: Evaluation-set and configuration selection

As reported in Section 3.1 and Appendix A.2, our results use the full official 500-task WebShop test split and ALFWorld's official valid-seen (140 games) and valid-unseen (134 games) splits. These evaluation splits are strictly disjoint from the training data and are used only for reporting results, not for selecting hyperparameters or checkpoints. A single configuration is fixed before the main evaluations and shared across benchmarks and model scales; every method is trained for 160 updates and reports the final checkpoint. Appendix B.2 varies the handoff threshold ζ\zeta and increment η\eta only to evaluate sensitivity; these results are not used for configuration or checkpoint selection.

Response to Weakness 1 and Question 1: Matched GiGPO and GiGPO–OPD controls

Following the reviewer's suggestion, we rerun the complete standard GiGPO method under exactly the same settings as TIDE and use it as the matched GiGPO baseline. We also evaluate GiGPO + OPD with fixed 1:11{:}1 coefficients and with Linear Handoff to test whether simply combining process-level credit with teacher supervision accounts for TIDE's gains. Our ATOD reproduction follows the official method, including trajectory-level GRPO and complete T-DUR.

MethodWebShop SR ↑\uparrowWebShop Score ↑\uparrowALFWorld Seen ↑\uparrowALFWorld Unseen ↑\uparrow
GRPO62.8±2.062.8\pm2.082.4±1.282.4\pm1.258.8±0.858.8\pm0.847.8±2.647.8\pm2.6
Matched GiGPO67.8±1.967.8\pm1.984.1±1.484.1\pm1.483.3±1.183.3\pm1.181.1±1.681.1\pm1.6
GiGPO + OPD (Fixed 1:1 Mixture)70.6±1.770.6\pm1.785.1±1.485.1\pm1.484.3±1.484.3\pm1.482.1±2.082.1\pm2.0
GiGPO + OPD (Linear Handoff)69.6±1.669.6\pm1.684.6±1.284.6\pm1.284.8±2.384.8\pm2.383.6±2.283.6\pm2.2
TIDE77.2±1.8\mathbf{77.2\pm1.8}89.8±1.2\mathbf{89.8\pm1.2}86.0±0.8\mathbf{86.0\pm0.8}85.1±1.5\mathbf{85.1\pm1.5}

Results. Complete GiGPO improves over GRPO, and both GiGPO–OPD controls further improve all four mean results, showing that process-level credit and teacher supervision are both useful. TIDE then exceeds the stronger GiGPO–OPD control on every metric: by 6.6 points on WebShop SR, 4.7 on WebShop Score, 1.2 on ALFWorld seen, and 1.5 on unseen success. These consistent gains demonstrate the additional effectiveness of TIDE's adaptive global–local coordination.

Response to Weakness 1 and Question 2: Schedule shape, transfer, tuning, and checkpoint selection

To separate delayed schedule shape from adaptive timing, we add Pilot-Matched Fixed Schedule ablations in which only the adaptive global handoff is replaced; the local mechanism and all other settings remain unchanged. The fixed windows are 51–114 for WebShop and 64–134 for ALFWorld, and we also transfer them across tasks without retuning.

MethodWebShop SR ↑\uparrowWebShop Score ↑\uparrowALFWorld Seen ↑\uparrowALFWorld Unseen ↑\uparrow
Linear Handoff72.2±2.272.2\pm2.286.3±1.386.3\pm1.385.7±1.985.7\pm1.981.3±2.081.3\pm2.0
TIDE77.2±1.8\mathbf{77.2\pm1.8}89.8±1.2\mathbf{89.8\pm1.2}86.0±0.8\mathbf{86.0\pm0.8}85.1±1.5\mathbf{85.1\pm1.5}
Pilot-Matched Fixed Schedule75.1±1.575.1\pm1.587.2±0.887.2\pm0.885.2±2.285.2\pm2.282.8±2.082.8\pm2.0
Fixed Schedule Transfer (WebShop →\rightarrow ALFWorld)------84.8±1.884.8\pm1.883.1±1.983.1\pm1.9
Fixed Schedule Transfer (ALFWorld →\rightarrow WebShop)74.9±1.574.9\pm1.586.9±1.186.9\pm1.1------

Results. The matched fixed schedule improves over the original full-budget schedules on some metrics, confirming that schedule shape contributes to performance. TIDE remains higher on all four metrics by 2.1, 2.6, 0.8, and 2.3 points, respectively, demonstrating an additional benefit from disagreement-based timing. Cross-task transfer lowers three of four metrics relative to the task-matched fixed schedules, and all transferred results remain below TIDE, showing that manually selected windows do not transfer reliably. TIDE instead determines the handoff from each task's training dynamics.

Response to Weakness 2 and Question 3: Standard deviations and per-seed success counts

We thank the reviewer for identifying this issue. Our audit found that the inconsistencies were confined to the transcription of several OPD and ATOD uncertainty values during table preparation; they do not affect the per-seed evaluations, mean results, any reported TIDE result, or the method rankings. Below, we provide the uniformly recomputed values together with the underlying integer success counts for each seed, and we will update Table 2 and all corresponding references in the main text and appendix.

For each success metric with NN tasks, we first compute the mean and sample standard deviation across the three seed-level integer success counts. We then divide both statistics by NN and multiply by 100 to report percentages.

ALFWorld overall success is computed from each seed's total successes over all 140 tasks, WebShop SR from 500 tasks, and WebShop Score from the three seed-level mean Scores.

ScalePick /35Look /13Clean /27Heat /16Cool /25Pick2 /24Overall /140
1.5B29, 32, 32; 88.6±4.988.6\pm4.98, 9, 9; 66.7±4.466.7\pm4.422, 22, 24; 84.0±4.384.0\pm4.312, 11, 12; 72.9±3.672.9\pm3.614, 16, 16; 61.3±4.661.3\pm4.615, 13, 13; 56.9±4.856.9\pm4.8100, 103, 106; 73.6±2.173.6\pm2.1
3B31, 31, 33; 90.5±3.390.5\pm3.39, 10, 10; 74.4±4.474.4\pm4.423, 25, 25; 90.1±4.390.1\pm4.312, 12, 13; 77.1±3.677.1\pm3.616, 16, 14; 61.3±4.661.3\pm4.612, 14, 12; 52.8±4.852.8\pm4.8103, 108, 107; 75.7±1.975.7\pm1.9
7B31, 32, 33; 91.4±2.991.4\pm2.910, 11, 12; 84.6±7.784.6\pm7.725, 26, 26; 95.1±2.195.1\pm2.112, 13, 13; 79.2±3.679.2\pm3.615, 17, 16; 64.0±4.064.0\pm4.015, 13, 13; 56.9±4.856.9\pm4.8108, 112, 113; 79.3±1.979.3\pm1.9
ATOD scalePick /35Look /13Clean /27Heat /16Cool /25Pick2 /24Overall /140
1.5B29, 31, 33; 88.6±5.788.6\pm5.79, 10, 11; 76.9±7.776.9\pm7.725, 26, 27; 96.3±3.796.3\pm3.714, 15, 16; 93.8±6.393.8\pm6.320, 21, 19; 80.0±4.080.0\pm4.015, 16, 14; 62.5±4.262.5\pm4.2112, 119, 120; 83.6±3.183.6\pm3.1
3B31, 34, 34; 94.3±4.994.3\pm4.99, 10, 11; 76.9±7.776.9\pm7.724, 27, 27; 96.3±6.496.3\pm6.413, 14, 15; 87.5±6.387.5\pm6.323, 21, 22; 88.0±4.088.0\pm4.017, 16, 15; 66.7±4.266.7\pm4.2117, 122, 124; 86.4±2.686.4\pm2.6
7B33, 34, 35; 97.1±2.997.1\pm2.910, 11, 12; 84.6±7.784.6\pm7.725, 26, 27; 96.3±3.796.3\pm3.715, 15, 15; 93.8±0.093.8\pm0.021, 22, 23; 88.0±4.088.0\pm4.018, 17, 16; 70.8±4.270.8\pm4.2122, 125, 128; 89.3±2.189.3\pm2.1
TIDE scalePick /35Look /13Clean /27Heat /16Cool /25Pick2 /24Overall /140
1.5B32, 34, 34; 95.2±3.395.2\pm3.39, 9, 10; 71.8±4.471.8\pm4.423, 24, 25; 88.9±3.788.9\pm3.716, 14, 15; 93.8±6.393.8\pm6.321, 22, 21; 85.3±2.385.3\pm2.318, 18, 16; 72.2±4.872.2\pm4.8119, 121, 121; 86.0±0.886.0\pm0.8
3B32, 33, 34; 94.3±2.994.3\pm2.99, 10, 11; 76.9±7.776.9\pm7.724, 25, 26; 92.6±3.792.6\pm3.715, 16, 14; 93.8±6.393.8\pm6.325, 23, 24; 96.0±4.096.0\pm4.019, 18, 17; 75.0±4.275.0\pm4.2124, 125, 126; 89.3±0.789.3\pm0.7
7B33, 33, 34; 95.2±1.695.2\pm1.611, 12, 13; 92.3±7.792.3\pm7.725, 26, 26; 95.1±2.195.1\pm2.116, 15, 16; 97.9±3.697.9\pm3.624, 24, 23; 94.7±2.394.7\pm2.319, 19, 18; 77.8±2.477.8\pm2.4128, 129, 130; 92.1±0.792.1\pm0.7
Table 3 (1.5B) methodWebShop SR /500ALFWorld Seen /140ALFWorld Unseen /134
GRPO303, 316, 32381, 83, 8360, 66, 66
Success Handoff343, 361, 36799, 105, 10594, 99, 101
Fixed 1:1 Mixture350, 364, 366106, 108, 110105, 107, 109
Linear Handoff349, 363, 371117, 121, 122106, 110, 111
Cosine Handoff358, 368, 372117, 119, 121107, 109, 111
w/o OPD Mod.364, 373, 379115, 118, 118106, 108, 110
w/o RL Mod.349, 369, 371116, 120, 121103, 107, 108
w/o Process Reward346, 366, 371109, 112, 115103, 106, 109
w/o Disagreement363, 374, 379115, 117, 119105, 107, 109
Additive Fusion315, 346, 35095, 101, 10191, 94, 97
TIDE376, 388, 394119, 121, 121112, 114, 116
Table 6 (3B) methodWebShop SR /500ALFWorld Seen /140ALFWorld Unseen /134
GRPO305, 320, 324103, 103, 10578, 81, 84
Success Handoff358, 375, 380108, 112, 113101, 104, 107
Fixed 1:1 Mixture364, 378, 381113, 116, 116105, 109, 109
Linear Handoff364, 380, 384120, 122, 124112, 112, 116
Cosine Handoff371, 382, 387119, 121, 123112, 114, 116
w/o OPD Mod.377, 389, 390120, 123, 123113, 114, 117
w/o RL Mod.364, 380, 384121, 123, 125110, 112, 115
w/o Process Reward361, 378, 383114, 118, 119108, 113, 113
w/o Disagreement377, 386, 392120, 122, 124111, 114, 115
Additive Fusion332, 357, 365101, 104, 10795, 98, 101
TIDE387, 396, 402124, 125, 126115, 117, 119

Response to Weakness 2 and Question 4: Table 1 quality grouping and all-turn analysis

The bottom and top 20% groups in Table 1 are determined solely by the GPT-5.5 process-quality score; the judge receives no outcome, future-trajectory information, teacher/student log-probabilities, or disagreement values. Ties at the group boundaries are resolved by uniform random sampling. We form the groups first and then compute their success rates and disagreement statistics. Thus, the observed 0% and 100% success rates are post-grouping results and are used neither for ranking turns nor for resolving ties.

To evaluate all turns rather than only the extremes, we add AUROC between judge scores 1 and 3 and Spearman correlation over all executable turns:

AnalysisIncluded turnsResult
Endpoint-quality discriminationJudge score 3 vs. score 1; score 2 excludedAUROC =0.4683=0.4683
All-turn ordinal associationAll executable turnsSpearman ρ=−0.0647\rho=-0.0647

Results. AUROC is close to chance, and the Spearman correlation is small in magnitude. In this analysis, disagreement has only a weak association with judge-rated process quality, suggesting that disagreement alone may be insufficient to consistently distinguish beneficial exploration from low-quality policy drift. This provides additional motivation for combining it with outcome-aware relative action value.

Response to Minor Suggestion (a): Figure 2 student scale and disagreement ranges

Both figures use the Qwen2.5-1.5B-Instruct student. Figure 2 and Table 1 measure disagreement on test-set trajectories, whereas Figure 3(c) measures it on on-policy rollouts from the training set. The different data distributions explain the different numerical ranges; the disagreement definition itself is unchanged. We will label this distinction explicitly in the captions.

Response to Minor Suggestion (b): Presentation and formatting corrections

We will remove the empty “Normalization pseudocode.” heading on page 16, correct the judge prompt's JSON formatting on page 14 so that the unintended LaTeX double backticks are not displayed, and replace the unexpanded macro in the OpenReview abstract with the rendered method name “TIDE.”

Reviewer_qmw4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TIDE, a method that coordinates on-policy distillation (OPD) and reinforcement learning (RL) for training multi-turn LLM agents. At the global level, TIDE gradually shifts weight from distillation toward RL when teacher and student disagreement decreases slowly. At the local level, it combines relative action values with teacher and student disagreement to combine both learning signals across interaction turns.

Experiments on WebShop and ALFWorld with Qwen2.5 students of different sizes show improvements over pure RL, pure distillation, and several hybrid baselines. The main contribution is an adaptive mechanism for balancing teacher guidance and reward-driven exploration throughout training.

Critical strengths

1:Table 1 provides a useful diagnostic: among high-disagreement turns, those rated as low and high quality have almost identical disagreement scores (0.189 vs. 0.188). This illustrates why disagreement alone cannot distinguish beneficial departures from poor policy drift and motivates incorporating an outcome-based value signal.

2:The local rule assigns distinct roles to the two signals: relative value determines the sign of the RL advantage, while disagreement scales its magnitude. Figure 4 provides a concrete illustration of the resulting allocation, with joint modulation placing greater weight on corrective “Go Back” actions. The local ablations show the same performance ordering at 3B (Table 6b), lending further support to this design.

Critical weaknesses
  1. The experiments do not isolate the benefit of disagreement-based timing from the shape of the resulting schedule. The closely related concurrent work RetireOPD [2] removes teacher supervision once disagreement stops shrinking and the student reaches a specified fraction of the teacher’s success rate. TIDE instead makes the transition gradual and retains a residual OPD term. RetireOPD is cited but not evaluated. More importantly, the Linear and Cosine baselines in Appendix B.2 spread the transition across the full training budget. By contrast, TIDE’s realized schedules in Figure 7 include an initial OPD-dominated plateau, a relatively concentrated transition, and a subsequent RL-dominated phase. The gains in Table 3a could therefore reflect this schedule shape rather than the use of disagreement to determine its timing. A fixed schedule that reproduces a comparable trajectory—ideally derived from independent pilot runs—would help distinguish these explanations.

  2. The local ablations and baseline comparisons leave the contribution of the local design unresolved. First, ATOD [1] already includes turn-level disagreement–uncertainty reweighting (T-DUR), which prioritizes distillation on turns with high disagreement or uncertainty. However, TIDE’s description of ATOD focuses on its linear schedule (Section 4 and Appendix A.1), without explicitly confirming whether the reproduction includes T-DUR. This detail matters for both the novelty of disagreement-based OPD weighting and the interpretation of the TIDE–ATOD performance gap. Second, TIDE uses a GiGPO-style step-level advantage [3], but the GiGPO results are taken from prior work, while the “w/o Process Reward” ablation changes both the RL advantage estimator and the OPD weighting. The gain over GRPO therefore cannot be attributed specifically to TIDE’s local coordination mechanism. This is particularly relevant at 1.5B on ALFWorld, where the cited GiGPO result is 86.7%, compared with TIDE’s 86.0%, although these cross-paper figures are not directly comparable. A matched GiGPO baseline and a GiGPO–OPD mixture with fixed coefficients would provide a clearer assessment.

References

[1] Qitai Tan, Zefang Zong, Mo Li, Yipeng Shi, Yang Li, and Peng Chen. “ATOD: Annealed Turn-Aware On-Policy Distillation for Multi-Turn Agentic Tasks.” arXiv preprint arXiv:2606.27814, 2026.

[2] Yan Yu et al. “RetireOPD: Self-Retiring On-Policy Distillation for Agentic Reinforcement Learning.” arXiv preprint arXiv:2609.20784, 2026.

[3] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. “Group-in-Group Policy Optimization for LLM Agent Training.” Advances in Neural Information Processing Systems, 38:46375–46408, 2025. DOI: 10.52202/085713-1544.

Questions for authors

1:What evidence distinguishes the benefit of disagreement-based timing from that of the resulting schedule shape? A fixed schedule derived from independent pilot runs, preserving the initial OPD-dominated plateau and concentrated transition, would help clarify whether adapting to the current run provides additional gains.

2:Does the reproduced ATOD baseline include its turn-level disagreement–uncertainty reweighting (T-DUR), in addition to linear annealing? Please clarify the implementation and any departures from the original method, as these details affect the interpretation of the TIDE–ATOD comparison.

3:The manuscript does not provide a code link or a reproducibility statement. Do you plan to release code?

4 comments

We sincerely thank the reviewer for the careful and constructive assessment. We particularly appreciate the recognition of Table 1's diagnostic motivation and of the distinct roles played by relative action value and teacher–student disagreement in our local coordination mechanism. Below, we respond to each concern separately.

Response to Weakness 1 and Question 1: Disagreement-based timing versus schedule shape

We acknowledge this limitation in the original ablation design. The Linear and Cosine ablations change the OPD–RL balance across the full training budget, whereas TIDE's realized handoff begins only midway through training. We therefore add a schedule-matched control to separate the delayed schedule shape from disagreement-based timing.

To isolate these factors, we add benchmark-specific Pilot-Matched Fixed Schedule ablations. We use the previously reported TIDE trajectories as pilots and freeze their realized transition windows before launching the new runs: updates 51–114 for WebShop and 64–134 for ALFWorld. These variants replace only TIDE's adaptive global handoff with the corresponding fixed schedule. The local value/disagreement-based weighting remains unchanged, and disagreement is no longer used to advance the global handoff. All other settings are also identical to TIDE. We additionally transfer each fixed schedule to the other benchmark without retuning to test whether a window derived from one task transfers to another.

Schedule policyWebShopALFWorld
TIDEOnline (realized: 51–114)Online (realized: 64–134)
Pilot-Matched Fixed ScheduleFixed: 51–114Fixed: 64–134
Fixed Schedule Transfer (WebShop →\rightarrow ALFWorld)—Fixed: 51–114
Fixed Schedule Transfer (ALFWorld →\rightarrow WebShop)Fixed: 64–134—

All methods follow the evaluation settings in Table 3.

MethodWebShop SR ↑\uparrowWebShop Score ↑\uparrowALFWorld Seen ↑\uparrowALFWorld Unseen ↑\uparrow
TIDE77.2±1.877.2\pm1.889.8±1.289.8\pm1.286.0±0.886.0\pm0.885.1±1.585.1\pm1.5
Pilot-Matched Fixed Schedule75.1±1.575.1\pm1.587.2±0.887.2\pm0.885.2±2.285.2\pm2.282.8±2.082.8\pm2.0
Fixed Schedule Transfer (WebShop →\rightarrow ALFWorld)——84.8±1.884.8\pm1.883.1±1.983.1\pm1.9
Fixed Schedule Transfer (ALFWorld →\rightarrow WebShop)74.9±1.574.9\pm1.586.9±1.186.9\pm1.1——

Results. The pilot-matched fixed schedule improves WebShop SR from 73.2% to 75.1% and ALFWorld unseen success from 81.3% to 82.8%, showing that the delayed schedule shape explains part of the gain. TIDE nevertheless remains higher by 2.1, 2.6, 0.8, and 2.3 points on the four metrics, respectively, supporting an additional contribution from disagreement-based timing. Under cross-task transfer, three of the four metrics decrease by 0.2–0.4 points, while ALFWorld unseen success increases slightly by 0.3 points; all transferred results remain below TIDE. These results suggest that fixed transition windows are task-dependent, whereas TIDE determines the handoff timing online from the disagreement dynamics of the current training run.

Response to Weakness 1: Relationship to RetireOPD

RetireOPD is closely related concurrent work: its first arXiv version appeared on September 17, 2026, immediately before the ICLR 2027 abstract deadline, and its revised version appeared after the full-paper deadline. We clarify the distinction below.

The methods differ at two levels. Globally, RetireOPD removes teacher supervision once disagreement stagnates and the student reaches a specified fraction of teacher performance; TIDE instead uses the disagreement-reduction trend to drive a gradual, monotone handoff, requires no teacher-relative performance threshold, and retains a residual OPD term. Locally, RetireOPD's adaptation is limited to this global retirement decision and does not introduce outcome-aware turn-level coordination between OPD and RL. TIDE additionally uses outcome-derived relative action value to provide signed turn-level RL credit, uses disagreement to scale that update, and jointly uses both signals to weight OPD across turns.

The table below compares the results reported in the RetireOPD paper with our TIDE results.

ModelMethodALFWorld SuccessWebShop ScoreWebShop SR/Acc
Qwen2.5-1.5BTIDE86.089.877.2
RetireOPD89.886.675.8
Qwen2.5-3BTIDE89.390.279.0
RetireOPD93.885.877.3
Qwen2.5-7BTIDE92.192.183.2
RetireOPD95.391.784.4

Results. Under their respective reported protocols, TIDE shows strong competitiveness on WebShop: it reports higher Score at all three model scales and higher success at 1.5B and 3B.

Response to Weakness 2 and Question 2: Complete ATOD reproduction and T-DUR

Our reproduced ATOD baseline follows the official paper and released code, including the complete T-DUR mechanism rather than an annealing-only variant. No ATOD-specific component is removed.

ATOD and TIDE differ at two levels. Globally, ATOD uses a predefined linear OPD-to-RL annealing schedule; TIDE instead uses the observed disagreement-reduction trend to drive the handoff online. Locally, ATOD's T-DUR combines teacher–student disagreement and student uncertainty only to reweight OPD, while its RL branch retains trajectory-level GRPO credit. TIDE instead uses outcome-derived relative action value for signed turn-level RL credit, uses disagreement to scale that update, and jointly uses relative value and disagreement to weight OPD across turns. Thus, ATOD adapts distillation priority, whereas TIDE coordinates both OPD and RL signals at the turn level.

For a controlled comparison, ATOD and TIDE use the same separately GRPO-trained and frozen teacher, with all non-method-specific training and evaluation settings held fixed. The teacher only provides token log-probabilities and is never jointly updated with the student.

Response to Weakness 2: Matched GiGPO and fixed GiGPO–OPD controls

Following the reviewer's suggestion, we rerun the complete standard GiGPO method under exactly the same settings as TIDE and use it as the matched GiGPO baseline. We also add a GiGPO + OPD control with fixed 1:11{:}1 coefficients to test whether simply combining process-level credit with teacher supervision accounts for TIDE's gains.

  1. Matched GiGPO: the complete standard GiGPO objective rerun under the TIDE protocol, without teacher supervision.
  2. GiGPO + OPD (Fixed 1:1): the same GiGPO objective combined with OPD using fixed 1:11{:}1 coefficients and uniform turn weights. This control uses neither TIDE's adaptive global handoff nor its local value/disagreement weighting.
MethodWebShop SR ↑\uparrowWebShop Score ↑\uparrowALFWorld Seen ↑\uparrow
GRPO62.8±2.062.8\pm2.082.4±1.282.4\pm1.258.8±0.858.8\pm0.8
Matched GiGPO67.8±1.967.8\pm1.984.1±1.484.1\pm1.483.3±1.183.3\pm1.1
GiGPO + OPD (Fixed 1:1 Mixture)70.6±1.770.6\pm1.785.1±1.485.1\pm1.484.3±1.484.3\pm1.4
TIDE77.2±1.8\mathbf{77.2\pm1.8}89.8±1.2\mathbf{89.8\pm1.2}86.0±0.8\mathbf{86.0\pm0.8}

Results. Matched GiGPO improves over GRPO, and adding fixed OPD provides further gains, confirming that both process-level credit and teacher supervision are useful. TIDE further exceeds the stronger GiGPO + OPD control by 6.6 points on WebShop SR, 4.7 points on WebShop Score, and 1.7 points on ALFWorld seen success. These consistent improvements show the additional effectiveness of TIDE's adaptive global–local coordination beyond process-level credit and fixed teacher supervision.

Response to Question 3: Code release and reproducibility

If the paper is accepted by the conference, we will release the TIDE code. We again thank the reviewer for all the valuable and constructive feedback above.

Thank you for the additional experiments. The pilot-matched control helps, but the remaining gap between TIDE and the fixed schedule is comparable to the reported standard deviations. For RetireOPD, the comparison does not clearly favor TIDE. RetireOPD performs better on ALFWorld at all scales. The matched GiGPO controls are helpful, and the WebShop gains are convincing. On ALFWorld, however, most of the improvement over GRPO already comes from GiGPO, and the remaining gap is small.

We are very grateful for your thoughtful follow-up and for taking the time to examine the additional experiments so closely. We have carefully considered both points and have clarified the corresponding discussion in the revision.

Pilot-matched control. We understand the concern. This control is deliberately conservative: it uses the exact window TIDE realized in its own run on the same task, so it is an oracle-style upper bound on fixed schedules. Against deployable preset schedules, TIDE's margins are 4.0 and 3.8 points on WebShop and ALFWorld valid-unseen success, well beyond seed variation. Against the oracle window, the direction is consistent on all four metrics (+2.1, +2.6, +0.8, +2.3), and the WebShop Score margin is about three times the pilot's standard deviation (2.6 vs 0.8). We agree that the ALFWorld valid-seen margin (0.8) is within seed noise. Transfer experiments further show that the same window moved to the other benchmark loses on three of four metrics and never reaches TIDE, so the gain comes from online timing rather than from a particular window. We state this in Section 3.3 and Appendix B.6.

RetireOPD. We appreciate this observation as well. RetireOPD is concurrent work. Its first arXiv version appeared on September 17, 2026, immediately before the ICLR 2027 abstract deadline, and its revised version appeared after the full-paper deadline. Under the ICLR 2027 reviewer guidelines, work published within two months of the full-paper deadline is considered contemporaneous, and comparisons with such work are encouraged but not required. We have nevertheless added a comparison in Appendix B.9 for completeness, which shows clear strengths of TIDE on WebShop. TIDE achieves higher WebShop Score at all three scales (89.8 vs. 86.6, 90.2 vs. 85.8, and 92.1 vs. 91.7) and higher WebShop success at 1.5B and 3B (77.2 vs. 75.8 and 79.0 vs. 77.3).

We hope these clarifications address your concerns, and we would be sincerely grateful if you would kindly consider updating your assessment to reflect the revised paper. If anything remains unclear, we would be very happy to provide further details.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.