- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The manuscript diagnoses a failure mode it terms intra-outcome-class reward-resolution collapse: under sparse binary team rewards, trajectories that share an outcome receive identical returns, so distinctions in task-relevant quality within success or failure are erased, and standard value learning cannot recover them from binary labels alone. TRACE addresses this by constructing a fixed terminal-state graph evaluator from the semantics of the original binary reward and the available terminal variables. The evaluator produces a within-class quality score, which is then propagated to preceding transitions through a discount-aware relabeling rule and exploited via prior-guided co-training with a decaying prior buffer and regularizer. The authors validate on SMAC v1/v2 and MPE Simple Spread, with ablations for the main modules, evaluator variants, prior count, and co-training hyperparameters.
Critical strengths
- The diagnostic study is the paper’s strongest asset. On 6h_vs_8z, the authors show substantial post-hoc quality variation among failed trajectories that all receive the same binary penalty, and they demonstrate that representative AIL, exploration, and offline-to-online methods recover that ordering only weakly or partially. This converts a vague complaint about sparse rewards into a measurable resolution problem, and it is the main reason the work merits serious consideration.
- The method is also simple, backbone-agnostic, and practically deployable. The evaluator is instantiated once and never queried during training; one evaluator is reused across all SMAC maps and another across all MPE settings. The ablation set is stronger than typical: TQR-Terminal isolates whether sparse terminal supervision alone suffices, while RandomTQR rules out the trivial explanation that any within-class variation helps. Gains are consistent across two action spaces and two backbones, and the prior-count study showing saturation at a few dozen trajectories is a useful practical finding.
Critical weaknesses
- The collapse result follows from the return definition, while the value and ordering results rely on assumptions that already assert the desired consistency and reduce to positive scaling; identifiability is therefore not established.
- No correlation is reported between evaluator scores and the post-hoc quality reference, so improved value alignment may reflect injected structure rather than recovered information; a length-correlated terminal bonus could yield similar results.
- The evaluator encodes health, alliance, proximity, and conflict priors, yet SimpleTQR already captures much of the gain and no misspecified or alternative evaluator is tested, leaving the graph contribution unclear.
- All non-terminal transitions receive the same value, so no within-trajectory credit assignment occurs and failed-trajectory returns become less negative with length; the scheme is not linked to potential-based shaping or return redistribution.
- Several baseline rows are identical across medium and medium-replay (LAIES in Table 3; MATD3, LAIES, CFCQL, MACal-QL in Table 4), with no significance testing, a short budget on hard maps, and TRACE behind ft-QMIX-DR on two dense-reward tasks.
- Prior-buffer admission and offline regularization are both evaluator-driven, so misspecification can be amplified; no diagnostic tracks drift from the reference distribution, and the assumed prior range is unverified on hard maps.
- Evaluation covers only SMAC v1/v2 and MPE Simple Spread, with no comparison to potential-based shaping, terminal-state intrinsic rewards, or return-redistribution baselines under the same prior budget, so gains may reduce to engineered terminal shaping.
Questions for authors
- What is the rank correlation between the evaluator’s own scores and the post-hoc quality reference, per map and per evaluator variant (LLM, SimpleTQR, RandomTQR)? Without this, what evidence supports recovery rather than construction?
- Since the relabeled reward is constant across all transitions of a trajectory, what is the precise mechanism by which the method produces transition-level rather than trajectory-level discrimination? Can you rule out the horizon-dependent incentive on failure trajectories?
- Tables 3 and 4 contain identical baseline entries across medium and medium-replay for LAIES, MATD3, CFCQL, and MACal-QL. Are these shared runs? If so, how do the two regimes differ for those methods?
- How does the relabeling scheme relate to potential-based shaping and episodic return redistribution? Is the ordering-preservation result anything beyond invariance to positive scaling?
- How was the performance range of the prior datasets verified on hard maps where the CFCQL “medium” policy may be close to random?
- Does the evaluator-based admission rule for the prior buffer bias the replay distribution toward the evaluator’s preferences, creating a feedback loop that amplifies evaluator misspecification? Do you have a diagnostic for this?
- What performance is obtained with a structured but deliberately misspecified evaluator, for example one with reversed within-class ordering, and what does this imply about the practical robustness of the stated assumptions?
Thank you for the clarifications and additional experiments. The explanations of baseline reporting and the matched comparisons are helpful. However, the weak or negative correlations between evaluator scores and the paper’s quality reference leave my main concern about within-outcome quality recovery unresolved. The noise experiments also do not directly address systematic misranking by the evaluator. I therefore maintain my rating of 2 (weak reject) and confidence of 2.