acceptodds
Under review as a conference paper at ICLR 2027

When Should Agents Differ? Learning to Schedule Behavioral Diversity from Training Dynamics in Multi-Agent RL

Abstract

Cooperative multi-agent reinforcement learning relies on parameter sharing for sample efficiency, and a family of diversity-promoting methods counteracts the resulting behavioral homogenization. All of these methods, however, apply their diversity signal at a strength that is fixed before training begins, leaving unanswered a question that logically precedes how to promote diversity: when should diversity be promoted, and how strongly, as training unfolds? We show empirically that this temporal dimension matters: on the same task, the diversity strength that helps early in training can actively harm late-stage coordination, and no constant strength is best across all of training. We formalize diversity scheduling as a meta-level sequential decision problem over training states, and derive a tractable solution, TIDE (Training-dynamics Informed Diversity schEduler). TIDE evaluates candidate diversity strengths offline via counterfactual branching from training checkpoints, distills the resulting preferences into a 1,300-parameter gate network, and deploys the gate online to reschedule the diversity strength from inexpensive training-log statistics at deployment-time overhead below 0.01% (one-time offline cost ≈175 GPU-h per algorithm–map pair, comparable to a coefficient sweep). Wrapped around three diversity-promoting algorithms (CIA, CDS, and LIIR), each with its own features, TIDE improves the base algorithm in all 18 algorithm-map combinations on six hard and super-hard SMAC maps, outperforming hand-designed decay schedules and an online bandit. The gate’s decisions admit post-hoc interpretation, recovering task-specific diversity schedules that align with each map’s tactical structure.

Primary area:
reinforcement learning
Keywords:
multi-agent reinforcement learning, cooperative MARL, behavioral diversity, adaptive scheduling, training dynamics, counterfactual evaluation, learning to rank, SMAC

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_LgpROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TIDE learns a small controller that adjusts the coefficient of an existing diversity mechanism (CIA, CDS, LIIR) during MARL training based on training statistics. The controller is supervised offline by branching five candidate coefficients from shared checkpoints and ranking their 200K-step outcomes. On six SMAC maps it improves the final win rate in all 18 algorithm/map settings, by 9.9 points on average over default coefficients, with extra results on SMACv2 and cross-map transfer.

Critical strengths
  1. The coefficient-ratio interface is simple and plugs into both auxiliary-loss and intrinsic-reward mechanisms, and the gain over the default is consistent across all 18 SMAC settings plus six SMACv2 scenarios.

  2. The authors include the right kinds of controls (tuned constant, time-only controller, frozen replay, budget-matched schedule search) and check how reliable the branch labels are (Table 2), which makes the evidence much easier to interpret.

Critical weaknesses
  1. The paper's central claim is that training feedback drives the schedule, but TIDE beats the time-only controller by only 0.7–2.5 points and budget-matched schedule search by −0.5 to +1.3 points, so what the results really support is "scheduling beats a fixed default".

  2. The 9.9-point headline compares a controller trained on 35M target-specific steps (more than 10 times a full training run) against untuned defaults, and the budget-matched comparison exists for only three CIA maps, where most of the gap disappears.

Questions for authors
  1. How does TIDE compare with population-based training of the same coefficient under an equal or smaller total step budget?

  2. Several base learners barely learn within 2M steps, are these consistent with published results, and do the gains hold with longer budgets?

  3. In Table 3, which source maps and how many source steps were used for each target, and does a time-only controller trained on the same source data transfer as well as TIDE?

4 comments

We thank the reviewer for the precise reading. The two weaknesses helped us state the contribution more exactly. Results obtained during the discussion period are reported in Tables R2 to R5 and are included in the revised PDF, where new text is marked in blue.

W1. Training feedback vs. scheduling per se

Table R1 decomposes TIDE's mean gain over the tuned constant (the best of our five candidates on development seeds) on the three maps of Table 1.

Table R1. Decomposition of TIDE's gain over the tuned constant, in points (computed from Table 1).

MapTuned → Time-onlyTime-only → TIDETIDE − Search
3s5z_vs_3s6z+4.2+2.5+1.3
2c_vs_64zg+0.8+0.7−0.5
6h_vs_8z+2.4+1.8+0.7

The time-only controller is built with the TIDE pipeline (same branch groups, targets, loss and stopping rule, App. E.2) and sees only t/T_max. It obtains 53 to 63% of the gain, which shows that the branch comparisons themselves produce useful schedules. Because the controls share TIDE's five deployment seeds, we also compared them pair by pair (15 pairs, stratified-bootstrap 95% CIs following Agarwal et al., 2021). TIDE is higher than time-only control in 11 of 15 pairs (+1.7 points, [−0.2, 3.6]) and than schedule search in 8 of 15 (+0.5, [−1.6, 2.5]). The revised Abstract, Introduction and Sec. 5.2 state that branch supervision accounts for most of the gain over fixed coefficients, that training statistics add further gains, and that TIDE performs comparably to budget-matched schedule search.

Training statistics matter most when the time course changes. Table R2 compares source-trained TIDE with a time-only controller trained on the same source data (protocol in Q3).

Table R2 (Table 3 in the revised PDF). Transfer with source-trained controllers (CIA, 5 seeds).

Target mapDefaultSource time-onlySource TIDE
5m_vs_6m.588.611.648
3s5z_vs_3s6z.248.271.316
6h_vs_8z.181.197.238

The gap between TIDE and time-only control is 0.7 to 2.5 points within a setting and grows to 3.7 to 4.5 points under transfer. Longer budgets (Q2) give a second test of the same kind.

W2. The 9.9-point headline vs. a 35M-step investment

35M steps equal 17.5 full 2M-step runs, and the revised Introduction states this next to the headline. It now reports three comparisons together, namely +9.9 points over default coefficients across 18 settings, +1.5 to +6.7 points over a tuned constant selected with 30M development steps on the three CIA maps of Table 1, and −0.5 to +1.3 points relative to budget-matched schedule search (34M). The Abstract summarizes the same comparisons.

For CIA, the default is a uniform α₀ = 0.02, the released value for 2c_vs_64zg and corridor. The CIA release tunes α per map for 5M-step training (0.1 on 3s5z_vs_3s6z, 0.005 on 6h_vs_8z). Our candidate set covers 0.005 (r = 0.25), and App. A.4 (Table 7) lists the released configurations next to ours.

Within one setting, TIDE's cost is comparable to tuning or search, and the corpus can additionally be reused on new maps. There the transferred gate needs no target-map development and reaches .316 on 3s5z_vs_3s6z and .238 on 6h_vs_8z, compared with .295 and .239 for a tuned constant that uses 30M target-map steps and .349 and .274 for search with 34M.

We also extended the tuned-constant control (5 candidates × 3 development seeds, then 5 held-out seeds) to two settings where the default is weak.

Table R3 (Table 17, App. E.6). Tuned constant on two further settings (5 seeds).

SettingDefaultTuned constantTIDE
CIA, MMM2.021.074.182
CDS, 2c_vs_64zg.003.097.183

TIDE remains 10.8 and 8.6 points above the tuned constant in these two settings.

Q1. PBT at an equal or smaller budget

We ran PBT over R = {0, 0.25, 0.5, 1, 2} with TIDE's cold start and P = 8. Every 100K steps, the bottom 25% copy weights and ratio from the top 25% and move the ratio to an adjacent value in R. One population costs 16M steps, less than TIDE's 35M one-time collection. We ran three independent populations per map and report both the best and the mean member.

Table R4 (Table 16, App. E.5). PBT vs. TIDE (CIA, PBT averaged over 3 populations, other methods over 5 seeds).

MapDefaultTuned constantPBT best memberPBT mean memberTIDE
3s5z_vs_3s6z.248.295.341.271.362
6h_vs_8z.181.239.262.207.281

TIDE is above both the best and the mean PBT member on both maps. The best member comes within about 2 points of TIDE, and the mean member stays below the tuned constant. The deployment cost also differs. With a trained gate, each new TIDE run is a single 2M-step learner, whereas PBT trains a population for each run.

Q2. Base-learner performance and longer budgets

All methods use SC2.4.10, difficulty 7 and 2M steps. The CIA release also uses SC2.4.10, with 2.05M steps on 2c_vs_64zg and 5.05M steps on 3s5z_vs_3s6z, 6h_vs_8z and corridor. At 2M steps, the CIA paper's curves are close to our defaults on 6h_vs_8z (about 0.2 vs. .181) and corridor (about 0 vs. .004) and slightly higher on 2c_vs_64zg (about 0.9 vs. .842). On 3s5z_vs_3s6z, the released α = 0.1 reaches about 0.5 at 2M. The CDS release uses our 2.05M budget on MMM2 and corridor, and its paper reports learning curves without numerical values. LIIR's original evaluation covers 3M, 8M, 2S3Z and 3S5Z, none of our maps, and an independent evaluation also reports that LIIR fails on corridor and 3s5z_vs_3s6z (Yang et al., AAMAS 2023). Table 7 (App. A.4) lists the budget and released coefficient next to our defaults.

Eight of the 18 defaults are below 0.05 (5 LIIR, 2 CIA, 1 CDS). In the other 10 settings the mean gain is 8.9 points, compared with 11.1 in these 8, so the gains hold across baseline levels.

The gate input excludes the training clock (Sec. 4.1), so a gate trained for 2M steps runs unchanged under a longer budget, whereas a time-indexed controller needs an extension rule. We rescaled t/T_max to the new budget.

Table R5 (Table 27, App. I.4). CIA at 5M steps with the frozen 2M-step gate (3 seeds).

MapDefaultTime-only (rescaled)TIDE (frozen gate)
MMM2.214.262.318
corridor.156.201.247

Beyond 2M steps the gate sees states outside its training range, and clipping affects 6.8% of queries in these runs, compared with 3.1% in the main runs. The frozen gate still leads rescaled time-only control by 5.6 and 4.6 points, more than the in-setting margins in Table 1.

Q3. Transfer protocol and time-only transfer

For each target map, the gate is trained on the pooled offline data of the other five CIA maps (125 checkpoint groups, 175M source steps). Feature statistics, clipping bounds and model selection use only source data, with validation regret on held-out source checkpoints, and no target-map interaction occurs before deployment. Sec. 5.5 now states this protocol and the source cost next to Table 3. The time-only controller in Table R2 uses the same source data, split, loss and stopping rule. Relative to the gain of target-trained TIDE over the default (.099 / .114 / .100), it recovers 16 to 23%, compared with 57 to 61% for source-trained TIDE.

We hope these revisions address the reviewer's concerns and would be glad to discuss further.

Reviewer_LgpROpenReview ↗

Your response only partially addressed my questions, but I am still willing to support your paper.

Given your responses to my questions and the revisions made to the main text, I have greater confidence in your paper. I have decided to increase your score.

We sincerely thank the reviewer for the constructive feedback and for raising the score. We acknowledge that some questions remain only partly resolved, and we will state these limitations clearly in the final version. Thank you again for your support.

Reviewer_UE9QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TIDE adapts diversity-incentive coefficients during cooperative multi-agent reinforcement learning. Offline branches compare five candidate coefficients from shared checkpoints over 200K steps, providing listwise supervision for a small controller. During new training runs, the frozen controller uses training statistics to select a coefficient every 20K steps. The interface supports auxiliary-loss and intrinsic-reward mechanisms without changing the agent architectures. Experiments with CIA, CDS, and LIIR on six SMAC maps report higher mean final win rates in all 18 settings, averaging 9.9 percentage points over default coefficients. Additional evaluations include SMACv2, CIA cross-map transfer, scheduling controls, behavioral measurements, and horizon/terminal-continuation diagnostics.

Critical strengths
  1. The paper addresses an important question in cooperative MARL: how diversity incentives should be adjusted as agents learn to coordinate.

  2. TIDE provides a flexible scheduling interface for auxiliary-loss and intrinsic-reward mechanisms without modifying the underlying agent architectures.

  3. The experiments cover three MARL algorithms and six SMAC maps, with improvements over default coefficients across all 18 settings. Additional SMACv2 and cross-map transfer experiments further support its effectiveness in the evaluated environments.

  4. The comparisons with alternative scheduling strategies, together with the supervision-horizon and policy-divergence analyses, help characterize the behavior of the proposed framework.

Critical weaknesses
  1. My main concern is whether training-state feedback provides a meaningful advantage over simpler scheduling strategies. Although TIDE consistently improves over default coefficients, the gains over stronger controls are much smaller. In Table 1, the improvements over the time-only controller are only 0.7–2.5 percentage points across three CIA maps. Compared with budget-matched schedule search, TIDE performs better on two maps but worse on the third, with differences ranging from -0.5 to +1.3 percentage points. Given the five-seed evaluation and the reported variability, I am not convinced that these differences establish a robust benefit from state-dependent feedback. The results seem to support diversity scheduling in general, but provide weaker evidence for the necessity of the proposed feedback controller.

  2. The methodological novelty also appears somewhat limited. TIDE learns to schedule an existing training coefficient using offline comparisons and listwise supervision. It remains unclear what substantial methodological advantage this design offers beyond existing adaptive hyperparameter scheduling approaches.

  3. Constructing the offline supervision dataset requires approximately 35M environment steps (about 175 GPU-hours) per algorithm–map setting. While the controller itself is inexpensive and its cost can be amortized through reuse, the practical benefit is difficult to justify given the relatively small improvements over budget-matched schedule search.

  4. The paper measures policy divergence, but I am not convinced that these measurements establish more effective agent specialization. Changes in policy divergence may reflect optimization effects rather than the emergence of complementary behaviors.

  5. Most controllers are trained separately for each algorithm–map setting. The CIA cross-map transfer results are encouraging, but it remains unclear whether the same controller can generalize across different diversity mechanisms or substantially different tasks without additional offline supervision.

Questions for authors

Q1. How robust are the improvements over the time-only controller and budget-matched schedule search? Could the authors provide paired confidence intervals or statistical tests for these comparisons, particularly on maps where the reported differences are below two percentage points?

Q2. Given the substantial offline collection cost and the comparable interaction budget of the schedule-search baseline, under what practical conditions does TIDE offer a clear advantage over direct schedule optimization? Can the authors quantify the cost-normalized benefit when accounting for controller reuse across multiple runs or tasks?

Q3. How can the authors distinguish useful agent specialization from an increase in policy divergence that does not improve coordination? Could additional behavioral or trajectory-level evidence establish a stronger connection between the learned schedules, complementary agent roles, and final task performance?

Q4. The paper examines supervision horizons of different lengths, but the ranking consistency between the primary horizon and longer horizons is only moderate in some settings. How sensitive are the learned schedules and final performance to the choice of the 200K-step supervision horizon relative to the 20K-step online decision interval?

Q5. Can a controller trained on one map or diversity mechanism generalize to a different algorithm or an unseen task without collecting target-specific counterfactual branches? In particular, how far can the existing cross-map transfer results be extended beyond CIA and the three evaluated maps?

4 comments

We thank the reviewer for a careful and fair review. The comments helped us state the claims more precisely. New analyses and experiments from the discussion period are reported in Tables R1 to R3 and are included in the revised PDF, where new text is marked in blue.

W1/Q1. Robustness of the gains over time-only control and schedule search

Because the scheduling controls share TIDE's five deployment seeds, we compared them pair by pair (15 pairs over three maps, stratified-bootstrap 95% CIs following Agarwal et al., 2021).

Table R1 (Table 15, App. E.4). Paired comparison with the scheduling controls (CIA, points).

TIDE vs.Pairs with TIDE higherMean paired difference [95% CI]
Time-only11/15+1.7 [−0.2, 3.6]
Frozen replay12/15+2.4 [−0.1, 4.9]
Schedule search8/15+0.5 [−1.6, 2.5]

TIDE is higher in 11 of 15 pairs against time-only control and in 12 of 15 against frozen replay, and it is comparable to budget-matched search. Sec. 5.2 now reports these paired results directly, and the phrase "consistent mean improvement when scheduling uses training feedback" has been removed.

Scheduling the diversity strength has strong support. For CIA, 24 of 30 matched seed pairs improve over the default (+10.7 points, 95% CI [7.0, 14.4], Wilcoxon p ≈ 9e-5), and time-only control, replay and search also exceed the default on all three maps. State input separates clearly from elapsed time when the controller is reused. Trained on the same source data, a time-only controller transfers 3.7 to 4.5 points worse than TIDE on the three target maps of Table 3, compared with 0.7 to 2.5 points within a setting (W3).

W2. Novelty relative to adaptive hyperparameter scheduling

Checkpoint branching and listwise ranking are established tools, and our contribution lies in how they are combined for diversity scheduling. Candidate strengths are compared from the same restored training state, the comparisons become supervised labels, and the resulting state-conditioned controller is reused across seeds and maps without further search.

This differs from existing approaches in what is learned and what is reused. PBT and PB2 adapt the coefficient within a concurrently trained population, and the resulting schedule is indexed by time. Our time-only and replay controls test that alternative directly. Learned schedulers such as RL learning-rate controllers (Xu et al., 2019), latent-ODE schedules fitted to sweep runs (Wiemann & Melchior, TMLR 2026) and AutoMBPO are fitted to runs that differ in their whole schedule, whereas TIDE's branches hold the restored state fixed and vary only the coefficient. Meta-gradient methods adapt a coefficient through a short differentiable lookahead. For intrinsic-reward weights (CDS, LIIR), much of the effect acts through changed data collection, which TIDE's 200K-step branches measure directly. LIIR itself meta-learns its intrinsic reward, and TIDE schedules its weight on top. Concurrent work posted after our submission (ADiCo, arXiv 2610.18808) tunes a DiCo diversity target online from returns, and the revised related work discusses it.

The learned schedules show suppression followed by reactivation on 3s5z_vs_3s6z and amplification followed by attenuation on 2c_vs_64zg. The linear and step decays in Table 1 cannot express these patterns.

Response to Reviewer UE9Q (2/2)

W3/Q2. When the offline cost pays off relative to schedule search

Within a single setting, TIDE and budget-matched schedule search reach similar means at similar budgets, and the revised text states this comparison directly. The offline corpus brings an additional benefit on new maps. Table R2 uses gates trained on the other five CIA maps (leave-one-map-out, 175M source steps collected once). All methods additionally train the deployed 2M-step run.

Table R2 (Table 14, App. E.3). Performance on a new map against target-map development cost (CIA, 5 seeds).

MethodTarget-map development steps3s5z_vs_3s6z6h_vs_8z
Default0.248.181
Source-trained TIDE0.316.238
Tuned constant30M.295.239
Budget-matched search34M.349.274

Without any target-map development, the transferred gate recovers 67% and 61% of search's gain over the default and reaches the level of a tuned constant that uses 30M target-map steps. Under the same transfer protocol, a time-only controller trained on the same source data is 3.7 to 4.5 points below source-trained TIDE on 5m_vs_6m, 3s5z_vs_3s6z and 6h_vs_8z.

W4/Q3. Policy divergence and useful specialization

JS divergence shows that identity-conditioned policies differ. To test whether the differences matter for the team, we ran two evaluation-only tests on the final checkpoints of the three CIA maps. Sec. 5.4 now reports them alongside the divergence curves.

Table R3 (Table 23, App. G.1). Functional tests (CIA, 5 seeds).

TestDefaultTIDE
Win-rate drop after identity swap, 3s5z / 2c / 6h (points)2.9 / 1.6 / 2.47.8 / 4.1 / 6.3
Same, relative to the method's win rate12% / 2% / 13%22% / 5% / 22%
Share of damage taken by zealots, 3s5z_vs_3s6z63%71%
Share of damage dealt by stalkers, 3s5z_vs_3s6z51%58%

Permuting identity inputs among same-type agents costs TIDE agents a larger share of their win rate, so their behavior depends more on identity. On 3s5z_vs_3s6z, roles across unit types are also sharper, with zealots absorbing more of the damage and stalkers dealing more of it. Both tests and the same-type divergence curves show a clearer division of work under TIDE.

Q4. Sensitivity to W = 200K relative to Δ = 20K

With Δ fixed, W = 50K/100K/200K/400K gives .291/.334/.362/.366 on 3s5z_vs_3s6z, and the 400K branches from the last checkpoint are truncated at 2M. With W = 100K, 2c_vs_64zg and 6h_vs_8z reach .872 and .259, compared with .891 and .281 at 200K, which follows the same trend. With W fixed, Δ from 5K to 50K changes the final win rate by at most 1.8 points and degrades only at 100K.

The moderate τ_b(W, 4W) reflects that the preferred fixed coefficient can change between 200K and 800K, and τ_b also counts swaps among near-tied candidates. How a fixed-coefficient label relates to deployment, where the choice is revisited every 20K steps, is a separate question, and terminal continuation measures it directly. It applies a candidate for Δ, hands control to the gate, and gives regret of .015 to .046. On 6h_vs_8z, which has the weakest agreement and the largest regret, TIDE still improves the default by 10.0 points.

W5/Q5. Transfer across mechanisms and tasks

Within CIA, every map has now been held out once under Table 3's protocol. On the three additional maps, the transferred gate reaches .861 / .094 / .051 on 2c_vs_64zg / MMM2 / corridor (defaults .842 / .021 / .004) and recovers 39 to 45% of the target-trained gain, compared with 57 to 61% on the maps of Table 3. Across mechanisms, the 13 optimization, value and progress features and the ratio action space are shared. A gate trained on CIA 3s5z_vs_3s6z with these features and deployed without target-algorithm branches reaches .351 for CDS (default .308) and .029 for LIIR (.003) on that map, and .149 for CDS on 6h_vs_8z (.118), recovering 33 to 47% of the algorithm-specific gain (App. I). A single gate shared by all mechanisms is a natural next step.

We hope these revisions address the reviewer's concerns and would welcome further suggestions.

Thank you for the detailed response and additional experiments. The paired analysis and same-source transfer comparisons strengthen the evidence for training-state feedback, and the identity-swap experiment shows that agent identity matters for the learned policies.

My remaining concerns are about what the feedback adds and when it is worth the cost. The identity-swap experiment compares TIDE with the default fixed coefficient, not with the learned time-only schedule. It therefore does not establish whether the greater reliance on identity is specific to feedback or would also arise from learned scheduling alone. The transfer results also show that TIDE can avoid target-map development when source data are already available. They leave open when collecting and reusing that source data offers a better overall cost–performance trade-off than direct schedule search.

These concerns remain despite the improvements in the revision, so I maintain my original rating.

Thank you for the careful follow-up. Both points are well taken. The reviewer is right that the identity-swap test compares TIDE with the default; as it is evaluation-only, we will add the time-only controller in the final version. On cost, TIDE and schedule search are comparable within a single setting (35M vs. 34M steps), but a searched schedule is tied to its map and budget, whereas the corpus continues to serve new maps, budgets and algorithms (Tables 3, 26, 27). We will state this trade-off explicitly. Thank you again for the thoughtful engagement.

Reviewer_7a6QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TIDE, an adaptive diversity scheduling framework for cooperative MARL. TIDE trains a lightweight ranking controller using counterfactual continuations with different diversity coefficients from shared checkpoints. The controller then adjusts diversity strength based on training dynamics, including optimization, value, and performance statistics. Experiments across multiple algorithms and tasks demonstrate its effectiveness, supported by scheduling comparisons, supervision reliability analyses, behavioral measurements, and cross-map transfer evaluations.

Critical strengths
  1. Well-motivated research problem with clear novelty. Rather than addressing generic hyperparameter adaptation, this work focuses on the evolving balance between behavioral diversity and coordination in cooperative MARL. Treating diversity strength as a training-state-dependent control variable offers a meaningful perspective.
  2. Effective formulation of counterfactual supervision. By comparing training outcomes under different diversity coefficients from shared checkpoints, TIDE transforms a challenging online control problem into an offline supervised ranking task. This design is conceptually clear and interpretable.
  3. Modular design and algorithm compatibility. TIDE integrates with CIA, CDS, and LIIR through a unified coefficient scheduling interface without modifying their underlying architectures. Its effectiveness across auxiliary-loss and intrinsic-reward mechanisms suggests good extensibility.
Critical weaknesses
  1. High offline supervision cost. Despite the lightweight controller, generating supervision requires extensive counterfactual training, costing approximately 35M environment steps and 175 GPU-hours per algorithm–map setting. This substantial overhead raises concerns about practicality in resource-constrained or frequently changing tasks.
  2. Limited evidence of broad generalization. Most experiments train separate controllers for individual algorithm–map settings. Although cross-map transfer is demonstrated, the results primarily establish framework compatibility rather than broad controller generalization. Cross-algorithm transfer and zero-shot generalization to unseen tasks remain insufficiently explored.
  3. Restricted diversity control space. TIDE selects from five predefined diversity ratios, {0, 0.25, 0.5, 1, 2}, limiting fine-grained or continuous adaptation. The sensitivity to candidate-set size, range, and granularity is not systematically evaluated, leaving the adequacy of this discretization unclear.
Questions for authors
  1. Could the authors further quantify the trade-off between offline supervision cost and performance gains, particularly when reducing the number of counterfactual branches or training checkpoints?
  2. Could the authors evaluate controller transfer across unseen tasks and different base algorithms, particularly under zero-shot settings without target-task supervision?
  3. Could the authors investigate sensitivity to the candidate-set size, coefficient range, and discretization granularity to justify the current five-value design?
4 comments

We thank the reviewer for recognizing the problem formulation, the counterfactual supervision design and the modularity of TIDE. The experiments below were run during the discussion period. They are reported in Tables R1 to R4 and are included in the revised PDF, where new text is marked in blue.

W1/Q1. Offline cost and the cost-performance trade-off

Core collection consists of 5 mother runs (10M steps) and 125 branches of 200K steps (25M), about 175 GPU-h per setting. Surrogate selection and hyperparameter sweeps are reported separately in App. E.3, and the revised main text states the collection cost in Sec. 5.5 and in the Limitations paragraph. The cost is paid once and the frozen gate is then reused, so with N runs each run carries about 175/N GPU-h (8.75 GPU-h at N = 20, less than one 2M-step run). Mother runs are ordinary default-coefficient runs. Existing baseline runs whose checkpoints store the full training state (networks, optimizer, replay buffer and random states) can therefore serve as mother runs, which leaves 25M branch steps. Gate fitting takes about 14 CPU-minutes, and gate forward passes take less than 0.01% of training time.

The alternatives also cost interaction. A tuned fixed strength uses 30M steps, budget-matched schedule search uses 34M, and a 5-value × 5-seed sweep uses 50M. TIDE exceeds the tuned constant by 6.7/1.5/4.2 points and is within seed variability of schedule search (+1.3/−0.5/+0.7). Within one setting its cost is thus comparable to these alternatives, and the same gate can also be applied to new maps without any collection (W2).

The paper reports two cost axes on 3s5z_vs_3s6z. Branch horizons W = 50K/100K/200K/400K (about 81/113/175/281 GPU-h) give final win rates of .291/.334/.362/.366, and corpora of 1/2/3/5/8 mother runs give validation regret of .0412/.0271/.0193/.0147/.0138. Following the reviewer's suggestion, we also deployed gates trained on reduced corpora. They reuse subsets of the existing branches, and the step column gives the budget needed to collect each corpus from scratch.

Table R1 (Table 13 in the revised PDF). Cost and performance on 3s5z_vs_3s6z (CIA, 5 unseen seeds).

CollectionStepsGPU-hFinal win rate
Default (no TIDE)00.248±.118
1 mother run, 5 checkpoints, 5 candidates7M35.291±.107
3 mother runs, 5 checkpoints, 5 candidates21M105.341±.094
5 mother runs, 3 checkpoints, 5 candidates25M125.338±.096
5 mother runs, 5 checkpoints, 3 candidates {0, 1, 2}25M125.322±.101
W = 100K (in the paper)22.5M113.334±.092
Full TIDE (in the paper)35M175.362±.087

With three mother runs (60% of the steps), TIDE keeps 82% of its gain over the default. A single mother run still adds 4.3 points, and W = 100K keeps 75% of the gain at 64% of the steps. At the same 25M budget, keeping all five candidates and using fewer checkpoints gives a higher mean than the reverse, and App. E.3 now recommends this configuration for limited budgets.

W2/Q2. Generalization across tasks and algorithms

All main results use deployment seeds that were never used in offline collection. Cross-map transfer (Table 3) is zero-shot with respect to the target map. The gate is trained on the pooled data of the other five CIA maps (175M source steps, collected once), with source-only feature statistics, no target branches and no fine-tuning. It improves the default by 6.0/6.8/5.7 points on 5m_vs_6m / 3s5z_vs_3s6z / 6h_vs_8z and recovers 57 to 61% of the target-trained gain. The same protocol applies to the other three maps without new offline branches.

Table R2 (Table 25, App. I.2). Zero-shot transfer to the remaining CIA maps (5 seeds).

Held-out mapDefaultZero-shot gateTarget-trained TIDE
2c_vs_64zg.842±.131.861±.124.891±.118
MMM2.021±.018.094±.081.182±.129
corridor.004±.004.051±.047.124±.098

The transferred gate therefore improves over the default on all six CIA maps, and on these three it recovers 39 to 45% of the target-trained gain.

For cross-algorithm transfer, 13 of the 14 to 16 features (optimization, value and progress statistics) are defined for all three algorithms, and for LIIR the value statistics come from its critic. Removing the mechanism-specific features raises validation regret only from .0147 to .0218, less than removing value (.0275) or progress (.0392) features. We trained a gate on CIA 3s5z_vs_3s6z with the 13 common features and deployed it zero-shot to CDS and LIIR.

Table R3 (Table 26, App. I.3). Zero-shot transfer across algorithms with a gate trained on CIA, 3s5z_vs_3s6z.

SettingDefaultCIA gate (zero-shot)Algorithm-specific TIDE
CDS, 3s5z_vs_3s6z.308.351.400
LIIR, 3s5z_vs_3s6z.003.029.082
CDS, 6h_vs_8z (unseen map).118.149.208

The CIA gate improves the default in all three cases without any target-algorithm branches and recovers 33 to 47% of the algorithm-specific gain. The remaining gap reflects that the same ratio acts differently on an auxiliary loss and on an intrinsic reward.

W3/Q3. Candidate-set size, range and granularity

Each candidate needs its own branches, so cost grows linearly with the size of the set. Every added value costs 25 branches (5M steps, about 25 GPU-h) per setting, and a continuous action cannot be labeled exhaustively. In our 2M-step constant sweep on 3s5z_vs_3s6z (α₀ = 0.02), the best swept value is interior (.283 at r = 0.25, against .176 at r = 0 and .141 at r = 2), and r = 0 is selected in the suppress-then-reintroduce schedules. The CIA release tunes α per map for its 5M-step budget. On this map it uses α = 0.1 (r = 5 in our units), which lies outside our range and reaches about 0.5 at 2M steps in the published curve (App. A.4, Table 7). Table R4 varies the set, including an extension to r = 4. Subsets reuse existing branches, and extra steps are counted relative to the five-candidate corpus.

Table R4 (Table 22, App. F). Candidate-set sensitivity (CIA, 3s5z_vs_3s6z, 5 unseen seeds).

Candidate setSizeExtra stepsFinal win rate
{0, 1, 2}30.322±.101
{0, 0.5, 1, 2}40.351±.090
{0, 0.25, 0.5, 1, 2} (paper)50.362±.087
{0, 0.25, 0.5, 0.75, 1, 1.5, 2}710M.364±.088
{0, 0.25, 0.5, 1, 2, 4}65M.357±.093
Scalar-encoded gate, 21 ratios on [0, 2], step 0.1210.347±.095

Results are similar from 4 to 7 candidates. The 3-candidate set, which removes both intermediate ratios, is 4.0 points lower, and adding r = 4 brings no further gain. The scalar-encoded gate queried at 21 ratios, 17 of them unseen in training, reaches .347, so a finer grid did not help in this test. We therefore keep five values, since larger sets add collection cost without a clear gain.

Presentation

The revised PDF adds a notation table (App. J, Table 28) and moves the cost and reuse summary into the main text (Sec. 5.5). The protocol descriptions will be shortened in the final version.

We hope these additions address the reviewer's concerns and would be glad to provide further details.

Thank you for your detailed response and the additional experiments. I appreciate the authors' efforts to address my previous concerns, and I believe the new results have partially alleviated them.

However, I share the concerns raised by the other two reviewers. The key novelty of this paper lies in leveraging training-state feedback for adaptive diversity scheduling, whereas the current experiments mainly demonstrate the effectiveness of dynamic scheduling in general, without sufficiently establishing the necessity or added value of training-state feedback over simpler time-based scheduling strategies.

Therefore, I have decided to maintain my original rating.

Thank you very much for the thoughtful follow-up. We agree this is the central question, and in case it is helpful, we would like to point to the experiments that test it directly. The time-only controller uses the same supervision as TIDE but observes only elapsed time. TIDE is higher on all three maps (Table 1), and the gap widens when the controller is reused: 5.7–6.8 vs. 1.6–2.3 points over the default on unseen maps (Table 3), and a 4.6–5.6 point lead under a 5M-step budget (Table 27). We sincerely thank the reviewer for the time and care devoted to our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.