- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TIDE learns a small controller that adjusts the coefficient of an existing diversity mechanism (CIA, CDS, LIIR) during MARL training based on training statistics. The controller is supervised offline by branching five candidate coefficients from shared checkpoints and ranking their 200K-step outcomes. On six SMAC maps it improves the final win rate in all 18 algorithm/map settings, by 9.9 points on average over default coefficients, with extra results on SMACv2 and cross-map transfer.
Critical strengths
-
The coefficient-ratio interface is simple and plugs into both auxiliary-loss and intrinsic-reward mechanisms, and the gain over the default is consistent across all 18 SMAC settings plus six SMACv2 scenarios.
-
The authors include the right kinds of controls (tuned constant, time-only controller, frozen replay, budget-matched schedule search) and check how reliable the branch labels are (Table 2), which makes the evidence much easier to interpret.
Critical weaknesses
-
The paper's central claim is that training feedback drives the schedule, but TIDE beats the time-only controller by only 0.7–2.5 points and budget-matched schedule search by −0.5 to +1.3 points, so what the results really support is "scheduling beats a fixed default".
-
The 9.9-point headline compares a controller trained on 35M target-specific steps (more than 10 times a full training run) against untuned defaults, and the budget-matched comparison exists for only three CIA maps, where most of the gap disappears.
Questions for authors
-
How does TIDE compare with population-based training of the same coefficient under an equal or smaller total step budget?
-
Several base learners barely learn within 2M steps, are these consistent with published results, and do the gains hold with longer budgets?
-
In Table 3, which source maps and how many source steps were used for each target, and does a time-only controller trained on the same source data transfer as well as TIDE?
4 comments
Q1. PBT at an equal or smaller budget
We ran PBT over R = {0, 0.25, 0.5, 1, 2} with TIDE's cold start and P = 8. Every 100K steps, the bottom 25% copy weights and ratio from the top 25% and move the ratio to an adjacent value in R. One population costs 16M steps, less than TIDE's 35M one-time collection. We ran three independent populations per map and report both the best and the mean member.
Table R4 (Table 16, App. E.5). PBT vs. TIDE (CIA, PBT averaged over 3 populations, other methods over 5 seeds).
| Map | Default | Tuned constant | PBT best member | PBT mean member | TIDE |
|---|---|---|---|---|---|
| 3s5z_vs_3s6z | .248 | .295 | .341 | .271 | .362 |
| 6h_vs_8z | .181 | .239 | .262 | .207 | .281 |
TIDE is above both the best and the mean PBT member on both maps. The best member comes within about 2 points of TIDE, and the mean member stays below the tuned constant. The deployment cost also differs. With a trained gate, each new TIDE run is a single 2M-step learner, whereas PBT trains a population for each run.
Q2. Base-learner performance and longer budgets
All methods use SC2.4.10, difficulty 7 and 2M steps. The CIA release also uses SC2.4.10, with 2.05M steps on 2c_vs_64zg and 5.05M steps on 3s5z_vs_3s6z, 6h_vs_8z and corridor. At 2M steps, the CIA paper's curves are close to our defaults on 6h_vs_8z (about 0.2 vs. .181) and corridor (about 0 vs. .004) and slightly higher on 2c_vs_64zg (about 0.9 vs. .842). On 3s5z_vs_3s6z, the released α = 0.1 reaches about 0.5 at 2M. The CDS release uses our 2.05M budget on MMM2 and corridor, and its paper reports learning curves without numerical values. LIIR's original evaluation covers 3M, 8M, 2S3Z and 3S5Z, none of our maps, and an independent evaluation also reports that LIIR fails on corridor and 3s5z_vs_3s6z (Yang et al., AAMAS 2023). Table 7 (App. A.4) lists the budget and released coefficient next to our defaults.
Eight of the 18 defaults are below 0.05 (5 LIIR, 2 CIA, 1 CDS). In the other 10 settings the mean gain is 8.9 points, compared with 11.1 in these 8, so the gains hold across baseline levels.
The gate input excludes the training clock (Sec. 4.1), so a gate trained for 2M steps runs unchanged under a longer budget, whereas a time-indexed controller needs an extension rule. We rescaled t/T_max to the new budget.
Table R5 (Table 27, App. I.4). CIA at 5M steps with the frozen 2M-step gate (3 seeds).
| Map | Default | Time-only (rescaled) | TIDE (frozen gate) |
|---|---|---|---|
| MMM2 | .214 | .262 | .318 |
| corridor | .156 | .201 | .247 |
Beyond 2M steps the gate sees states outside its training range, and clipping affects 6.8% of queries in these runs, compared with 3.1% in the main runs. The frozen gate still leads rescaled time-only control by 5.6 and 4.6 points, more than the in-setting margins in Table 1.
Q3. Transfer protocol and time-only transfer
For each target map, the gate is trained on the pooled offline data of the other five CIA maps (125 checkpoint groups, 175M source steps). Feature statistics, clipping bounds and model selection use only source data, with validation regret on held-out source checkpoints, and no target-map interaction occurs before deployment. Sec. 5.5 now states this protocol and the source cost next to Table 3. The time-only controller in Table R2 uses the same source data, split, loss and stopping rule. Relative to the gain of target-trained TIDE over the default (.099 / .114 / .100), it recovers 16 to 23%, compared with 57 to 61% for source-trained TIDE.
We hope these revisions address the reviewer's concerns and would be glad to discuss further.
Your response only partially addressed my questions, but I am still willing to support your paper.
Given your responses to my questions and the revisions made to the main text, I have greater confidence in your paper. I have decided to increase your score.
We sincerely thank the reviewer for the constructive feedback and for raising the score. We acknowledge that some questions remain only partly resolved, and we will state these limitations clearly in the final version. Thank you again for your support.
We thank the reviewer for the precise reading. The two weaknesses helped us state the contribution more exactly. Results obtained during the discussion period are reported in Tables R2 to R5 and are included in the revised PDF, where new text is marked in blue.
W1. Training feedback vs. scheduling per se
Table R1 decomposes TIDE's mean gain over the tuned constant (the best of our five candidates on development seeds) on the three maps of Table 1.
Table R1. Decomposition of TIDE's gain over the tuned constant, in points (computed from Table 1).
The time-only controller is built with the TIDE pipeline (same branch groups, targets, loss and stopping rule, App. E.2) and sees only t/T_max. It obtains 53 to 63% of the gain, which shows that the branch comparisons themselves produce useful schedules. Because the controls share TIDE's five deployment seeds, we also compared them pair by pair (15 pairs, stratified-bootstrap 95% CIs following Agarwal et al., 2021). TIDE is higher than time-only control in 11 of 15 pairs (+1.7 points, [−0.2, 3.6]) and than schedule search in 8 of 15 (+0.5, [−1.6, 2.5]). The revised Abstract, Introduction and Sec. 5.2 state that branch supervision accounts for most of the gain over fixed coefficients, that training statistics add further gains, and that TIDE performs comparably to budget-matched schedule search.
Training statistics matter most when the time course changes. Table R2 compares source-trained TIDE with a time-only controller trained on the same source data (protocol in Q3).
Table R2 (Table 3 in the revised PDF). Transfer with source-trained controllers (CIA, 5 seeds).
The gap between TIDE and time-only control is 0.7 to 2.5 points within a setting and grows to 3.7 to 4.5 points under transfer. Longer budgets (Q2) give a second test of the same kind.
W2. The 9.9-point headline vs. a 35M-step investment
35M steps equal 17.5 full 2M-step runs, and the revised Introduction states this next to the headline. It now reports three comparisons together, namely +9.9 points over default coefficients across 18 settings, +1.5 to +6.7 points over a tuned constant selected with 30M development steps on the three CIA maps of Table 1, and −0.5 to +1.3 points relative to budget-matched schedule search (34M). The Abstract summarizes the same comparisons.
For CIA, the default is a uniform α₀ = 0.02, the released value for 2c_vs_64zg and corridor. The CIA release tunes α per map for 5M-step training (0.1 on 3s5z_vs_3s6z, 0.005 on 6h_vs_8z). Our candidate set covers 0.005 (r = 0.25), and App. A.4 (Table 7) lists the released configurations next to ours.
Within one setting, TIDE's cost is comparable to tuning or search, and the corpus can additionally be reused on new maps. There the transferred gate needs no target-map development and reaches .316 on 3s5z_vs_3s6z and .238 on 6h_vs_8z, compared with .295 and .239 for a tuned constant that uses 30M target-map steps and .349 and .274 for search with 34M.
We also extended the tuned-constant control (5 candidates × 3 development seeds, then 5 held-out seeds) to two settings where the default is weak.
Table R3 (Table 17, App. E.6). Tuned constant on two further settings (5 seeds).
TIDE remains 10.8 and 8.6 points above the tuned constant in these two settings.