- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies how to improve a frozen multi-agent policy without retraining it. At each decision point, the method generates small joint residual actions around the native IPPO action. GPE predicts which candidate can make better short-horizon progress toward the goal, and then selects that residual or keeps the original action. The main idea is that predicting a local action effect may be easier for action selection than predicting the final episode return. The experiments mainly use the VMAS Transport task and show clear improvements over native IPPO and Direct Completion.
Critical strengths
-
The basic motivation makes sense. Small residual actions may only have limited short-term effects, so predicting the full episode return can make their differences difficult to identify.
-
The comparison with Direct Completion is quite good. The authors keep the candidate bank, input features, architecture and training budget the same, and mainly change the prediction target. This makes the improvement easier to attribute to the target choice.
-
The main IPPO results are strong and consistent. Guarded and greedy GPE improve return much more than Direct Completion, and the random residual control performs poorly. The authors also report native-success losses instead of only showing average return.
Critical weaknesses
-
I am not fully convinced about the novelty of the GPE target in this task. The paper shows that goal progress is almost exactly the four-step reward advantage under the Transport reward. So GPE can also be understood as predicting a short-term reward advantage for residual action selection, which seems much less novel than the current paper suggests.
-
The paper does not provide the most important baseline for this interpretation. I would like to see a simple predictor directly trained on the same short-term reward advantage, using the same features and architecture. Since the paper shows this target is mathematically equivalent to GPE under the current reward, I'm not sure what benefit specifically comes from the proposed goal-progress formulation.
-
The experiments are not convincing enough. Most strong results come from one VMAS Transport task with IPPO. On MAPPO and ISAC, GPE does not outperform Direct Completion, which makes the generality of the main argument questionable.
-
The method relies alot on task-specific information. The scorer uses hand-designed geometric features about the package, goal and agents, and the progress target directly uses package-to-goal distance. The raw-input ablation performs much worse, so Im not sure how easily the method can move to tasks without an obvious progress metric or geometry.
Questions for authors
As in weaknesses.
4 comments
W2:
We agree that a matched short-horizon reward predictor is the key control. We therefore added R4, using the same frozen backbone, candidate bank, features, architecture, data, and selection rule as GPE; only the prediction target differs. On Transport, the two targets are nearly identical (MAE ), and their closed-loop performance is also similar: GPE obtains return versus for R4, with a paired difference of and 95% CI . We therefore do not claim a unique geometric advantage on Transport.
We additionally evaluate GPE and R4 on Discovery, Dispersion, and Sampling, where progress and short-horizon reward differ. GPE improves over R4 across all tested task-backbone settings, supporting the benefit of task-informed progress supervision when the two targets are not equivalent.
The distinction is the supervision supplied by the task-progress potential. In Discovery, agents can approach an unfinished coverage target without triggering a reward event during the prediction horizon. Dispersion similarly separates approach to an unvisited landmark from its first-visitation reward. In Sampling, density-weighted remaining travel distance differs from newly sampled density. GPE can therefore distinguish candidates through their geometric endpoint effects even when their short-horizon rewards provide little differentiation. The matched experiments test whether learning this additional supervision translates into higher native-task return; they do not give GPE extra deployment inputs or change the environment reward.
We accordingly defend the framework through the controlled target comparison: its Transport behavior is consistent with the known reward identity, while its additional-task results demonstrate that progress-based supervision can improve the same residual-selection mechanism when progress and reward differ. The claim is a useful task-informed choice of supervision for post-training joint action selection, not that equivalent targets must produce different information or that GPE universally dominates reward prediction.
W3:
We agree that the original evidence was strongest for Transport with IPPO. To address this, we expanded the evaluation beyond a single task and backbone, see W1 response. In addition to the existing Transport backbone study, we added experiments on three further VMAS tasks—Discovery, Dispersion, and Sampling—with multiple frozen backbones, including IPPO, MAPPO, and MADDPG. These additional results broaden the evidence substantially beyond the original Transport/IPPO setting. We will revise the paper accordingly and frame the contribution as a post-training residual-selection mechanism that can be applied across multiple cooperative tasks and frozen backbone policies, rather than claiming uniform improvement over every comparator in every setting.
W4:
We agree that the current GPE implementation is task-informed: its progress potential and input representation use task structure. The raw-input ablation shows that the chosen structured representation matters under the tested training budget; it does not establish that GPE automatically discovers a suitable representation. We will make this assumption explicit. The additional experiments clarify how much adaptation is required. We use existing VMAS Discovery, Dispersion, and Sampling scenarios with native rewards unchanged. We define their progress potentials ourselves: remaining two-agent coverage deficits in Discovery, distance deficits to unvisited landmarks in Dispersion, and density-weighted remaining distance to unsampled cells in Sampling. These are not built-in VMAS supervision targets. The task-specific work is to specify the potential, adapt the observation-derived input representation, and collect counterfactual branches to fit new scorers. The candidate-relative target construction, bounded joint residual mechanism, frozen-backbone interface, and greedy candidate-selection procedure are retained. These experiments therefore demonstrate applicability with explicit task adaptation, rather than zero-shot transfer of one scorer. The appropriate scope is task-informed post-training intervention: the reusable component is the joint residual-selection mechanism, while an informative progress metric and new scorer fitting are required for a new task.
The positive matched-target results on three additional scenarios and multiple frozen backbones, reported in W1, show that the framework is not restricted to package-to-goal distance. They support its use when an informative progress potential can be specified and counterfactual training labels can be collected. Learning an appropriate progress representation is an extension to investigate, rather than a capability demonstrated by the present experiments. Full definitions and controls are provided in Appendix Section M.
Thank you for the response
Very grateful for the rebuttal and the additional experiments. I appreciate the authors' efforts. But I still have concerns about the novelty. The proposed framework mainly combines task-specific progress prediction with existing residual action selection techniques. The new results show that carefully designed progress targets can improve performance over short-horizon reward prediction, but I am not convinced that this represents a sufficiently novel methodological contribution. The proposed approach also still relies heavily on manually designed task-specific progress potentials and input features, which limits its general applicability. I acknowledge the improved experimental validation, but I will maintain my original score.
We appreciate the reviewer’s acknowledgement of the additional experimental validation, and we would like to clarify the intended methodological contribution and the extent of task-specific design more carefully.
Our contribution is in formulating and validating candidate-relative, task-informed short-horizon prediction for post-training complete-team residual selection around a frozen multi-agent policy. GPE changes how the correction is selected. Standard residual RL learns a residual policy directly by optimizing reward, while predictive-control approaches choose corrections through rollouts under a learned dynamics model. GPE instead keeps the cooperative backbone frozen and learns a scorer for the relative short-horizon effect of each bounded complete-team intervention compared with the native-policy branch from the same state. For candidate (q),
where (q=0) is the frozen native-policy reference and (q) applies a bounded joint residual. At deployment, GPE does not learn a new policy or roll out a learned world model, it directly ranks candidate joint corrections using the learned candidate-relative score. The matched R4 comparison uses the same scorer architecture, inputs, candidate bank, branch data, fitting budget, and selection rule, thereby isolating the supervision target.
We also agree that the current method is task-informed, but we would like to clarify the extent of manual design. The progress potentials are constructed from task quantities and event rules already defined by the VMAS scenarios, rather than introducing new environment objectives or modifying the native rewards. The task-specific quantities are largely inherited from the existing VMAS scenario definitions rather than introduced arbitrarily by us. For example, Discovery already defines the two-agent coverage requirement and the (0.25) coverage radius, while Dispersion already defines the single-agent visitation rule and the (0.055) contact threshold. Sampling provides the density field and visitation state, and Transport already uses package-to-goal distance in its native shaping. We use these existing scenario quantities to construct a short-horizon progress potential; we do not change the native rewards, task objectives, or event thresholds. The task-specific design consists of expressing an intermediate progress measure from existing scenario quantities and fitting a new scorer. The paired-rollout construction, candidate-relative four-step target, bounded complete-team residual interface, frozen-policy reference, and candidate-selection procedure remain unchanged across tasks.
Regarding general applicability, the revised evaluation extends beyond the original Transport setting to three qualitatively different cooperative VMAS tasks Discovery, Dispersion, and Sampling with multiple frozen backbones. These tasks involve different objectives and reward structures, including multi-agent target coverage, first visitation of landmarks, and density-weighted spatial sampling. Across these settings, GPE improves over the corresponding frozen native policy and also outperforms the matched R4 predictor; the additional experiments therefore provide evidence that the same residual-selection formulation can be instantiated beyond Transport and IPPO.
W1. The paper shows that goal progress is almost exactly the four-step reward advantage under the Transport reward. So GPE can also be understood as predicting a short-term reward advantage for residual action selection, which seems much less novel than the current paper suggests.
We thank the reviewer for identifying this distinction. We agree that, under the conditions stated in the paper, the Transport GPE target is effectively equivalent to the four-step reward advantage. Our contribution should therefore not be interpreted as a mathematically new target in this special case. We will clarify the connection to short-horizon advantage prediction and residual control.
The contribution we investigate is candidate-relative future progress as supervision for post-training joint residual selection: the scorer estimates the effect of a bounded joint intervention relative to the frozen policy, and the selector uses those estimates to choose among candidate actions. The relevant empirical question is whether task-informed progress supplies useful supervision beyond the native short-horizon reward. We therefore added the requested four-step reward predictor (R4), matched to GPE within each task–backbone setting in input features, architecture, counterfactual branch data, initialization, fitting budget, candidate bank, and selection rule; only the supervised target differs.
To examine the distinction beyond this special case, we evaluated existing VMAS Discovery, Dispersion, and Sampling scenarios with native rewards unchanged. We defined task-informed progress potentials for these scenarios; they are not built-in VMAS targets. Coverage and first-visitation rewards can remain unchanged while agents approach unfinished objectives, and Sampling rewards newly sampled density rather than reduction in density-weighted remaining travel distance. Thus, the resulting progress labels need not equal four-step reward advantages. The post-selection mechanism is retained, while task-specific input adapters and scorers are fitted for each setting; this is applicability across tasks and backbones
Across the five additional task–backbone settings reported below, GPE improves return over the matched R4 predictor. The direct GPE-minus-R4 gains are +0.0594 on Discovery/IPPO (95% CI [0.0432, 0.0740]), +0.4320 on Dispersion/MAPPO ([0.3507, 0.4887]), +0.3658 on Dispersion/IPPO ([0.2472, 0.4861]), +2.1381 on Sampling/IPPO ([1.7430, 2.5866]), and +0.3072 on Sampling/MADDPG ([0.0836, 0.6853]). These comparisons support the benefit of task-informed progress supervision when it differs from the native reward target. They do not imply uniform superiority across all tasks or backbone policies.
R4 denotes the matched four-step reward predictor and DC denotes Direct Completion. Each setting uses 3 frozen policies, 3 scorer fits per policy, and 5,000 matched assignments with Q=33 and four-step greedy selection. CIs use hierarchical paired bootstrap resampling; SD is across backbone-level means, and gains use unrounded means.
The evidence therefore supports a specific claim: task-informed candidate-relative progress can be a more effective supervision target for joint residual selection than short-horizon reward advantage when the two encode different information. We will revise the novelty claim accordingly, while explicitly acknowledging the Transport equivalence and the method’s task-informed nature.