- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies constrained multi-agent control and proposes a method called VERA. The method exploits an asymmetry between training and execution: during training, the system can compute the exact feasibility of an action after it is proposed, whereas during execution it cannot. VERA introduces two training-time mechanisms: VFR (verifiable feasibility representation, a representation supervised by feasibility margins) and CGRA (counterfactual group-relative advantage, which uses a one-sided what-if evaluator to evaluate multiple representation–action candidates and applies group-relative normalization). In SAGIN experiments, under reward-matched conditions, VERA’s success rate is about 9 percentage points higher than the strongest reward-matched baseline in the paper (AB-MAPPO-R), while substantially reducing coverage violations. The experiments show that the performance gain comes mainly from CGRA, while the semantic content of the representation itself has little effect; in addition, in two other domains, CGRA runs into problems such as reward hacking and training collapse.
Critical strengths
-
The main strength of this paper is its solid analysis of where the performance gain comes from. With extensive comparisons, comprehensive ablations, and rigorous statistical tests, readers can clearly see which factors drive the improvement. In addition, the clear separation between training and execution ensures very low deployment overhead.
-
The authors honestly report unfavorable results and failure cases in the main text, and this honesty deserves credit.
Critical weaknesses
-
The performance gain is not disentangled from the use of extra information: CGRA has access to what-if outcomes of unexecuted representation–action candidates, and VERA receives an additional validity reward, neither of which is available to the baselines.
-
The gain from CGRA is observed only in a simplified custom simulator and fails in the other two domains; its applicability criterion is derived post hoc, yet the title and abstract imply general applicability, which is an overclaim.
Questions for authors
-
Can the baselines be given the same extra information, i.e., access to what-if outcomes of unexecuted representation–action candidates and the validity reward? If not, how can the gain be attributed to VERA rather than to the extra information?
-
How does VERA compare empirically to COMA and difference rewards, and what distinguishes CGRA from them conceptually?
-
Can the applicability criterion for CGRA be stated before the experiments and tested on a held-out domain? Otherwise, would the authors revise the title and abstract to reflect the limited scope?
-
Please specify the candidate sampling method, the candidate scoring function, and the hyperparameter settings.
6 comments
Response to Reviewer 1ETU — Part 2
Continued from Part 1. This part covers Q2 (COMA and difference rewards), W2/Q3 (scope and overclaiming), and Q4 (sampling, scoring, and hyperparameters).
Q2 — comparison with COMA and difference rewards
Our original COMA baseline implemented the baseline on the wrong variable: it held the agent's own executed action fixed and sampled peers. The COMA paper defines the baseline by marginalizing the agent's own action while holding peers at their executed actions. We reimplemented it accordingly — centralized Q(s, a_1 … a_K, i), M = 4 samples of the agent's own action, Q − b z-scored and blended exactly as CGRA, with G = 1 so that the advantage source is the only difference — and reran five seeds under the corrected physics.
| Arm | SR (%) | COV (%) | Local share |
|---|---|---|---|
| COMA (corrected, M = 4) | 53.30 ± 0.02 | 0.00 | 100% from episode 30 |
| CGRA (= M3, S2) | 53.65 ± 0.82 | 2.01 ± 0.17 | 94.9% |
| All-local control | 53.31 ± 0.21 | 0.00 | 100% |
The corrected baseline does not fail, but it does not learn to offload either. Coverage violations fall to exactly zero by episode 30 and stay there in all seeds; an instrumented replay shows the local share rising from 9.6% to 100.0% at episode 30; the converged SR matches the never-offload control to 0.01 points. The critic explains it: after correct marginalization, mean Q and mean b agree to seven decimals (143.594685 vs. 143.594688), so the advantage is numerically zero and its z-score carries no action-dependent signal. COMA converges to the conservative local policy — far better than the earlier incorrect run, but not evidence of selective offloading. CGRA reaches a comparable SR while offloading on 5.1% of decisions and without training a critic.
On difference rewards: a difference-reward baseline requires a per-agent difference function D_i = R − R_(−i), which in turn requires a decomposition of the global reward. The coverage term here is a shared constraint, not a separable sum of agent contributions, so we did not construct such a baseline and do not claim an empirical comparison. Conceptually, CGRA estimates the counterfactual from the policy's own candidates rather than from a designed reward decomposition, which is what allows it to use the simulator's per-candidate rewards and certificates directly. We state this as a conceptual distinction, not as an empirical result.
W2/Q3 — scope, post-hoc criterion, and overclaiming
We agree with the criticism. The gain is established in SAGIN and does not transfer to the two stress domains; the criterion in Section 5 was induced from those failures rather than stated in advance; and the earlier abstract and conclusion implied broader applicability than the evidence supports.
The revision rewrites the abstract and conclusion to scope the claim to constrained offloading of this class, states the criterion as an explicit, testable hypothesis before the experiments, and keeps the title unchanged because it already names the constrained setting. A held-out-domain test is the next step and is not claimed here. The all-local control further narrows the performance claim: in this benchmark, most of CGRA's benefit is avoiding harmful offloads rather than beating a local-only policy. Never offloading scores 53.31 ± 0.21% SR with 0.00% COV; selective offloading adds +3.33 points (p = 0.115). The revision states this bound explicitly.
Q4 — candidate sampling, scoring, and hyperparameters
- Sampling: G = 8 i.i.d. samples from π_θ(· | o_i) per agent per step, drawn in one batched pass; no top-k, no reranking; candidate 0 is executed.
- Decoding: each candidate maps to a node n_g, an offload ratio ρ_g, and a bandwidth request b_g.
- Scoring: r_exec(g) is the RLVR execution reward (success, latency, weighted energy, coverage violation); c*(g) is the five-dimensional certificate; r_total(g) = r_exec(g) + λ_v · [ −(1/5) || ĉ_i − c*(g) ||₁ ] + λ_c · [ −viol(a_g, ĉ_i) ], with λ_v = 0.5 and λ_c = 0.1.
- Advantage: A_i = clip( (r_total(0) − mean_g r_total) / (std_g r_total + 10⁻⁸), −3, 3 ), blended as 0.5 · GAE + 0.5 · A_i.
- Training: five seeds × 1,000 episodes, last 500 episodes as the reported window; the constrained baseline uses 3,000 episodes at the matched budget.
- Configuration: full hyperparameter snapshots are in the updated artifact under
rebuttal_evidence/configs/(experiment_configs.json,mechanism_ladder.yaml), with per-seed CSVs and the recomputation script in the same directory.
Appendix A.2 gives the same definitions and the pseudo-code; the implementation is in grpo_trainer.py (_grpo_group_step) and env/sagin_env.py (evaluate_candidates).
Thank you for the detailed responses and for the information-matched ladder, the released-code rerun, the corrected COMA run, Algorithm 1 and Table 14, which address much of my review. Numbers refer to the revised PDF. Three points remain open for me; none of them needs a new experiment.
-
W1/Q1 (where the gain comes from). The what-if half is answered within VERA: at the same 160 queries the within-group baseline adds +14.18 points on the released code (p. 15), and M1 shows no detected gain over M0 (Table 9). Still open: (a) p. 7 and p. 19 say the arms share a reward, but not whether M0 and M1 carry Eq. 6's λ_v and λ_c terms. Your reply's "validity term" −λ_c·viol is Eq. 5's consistency term, so the λ_v term (Q1's validity reward) remains open. (b) Table 9 and p. 7 call M2 a "global baseline", but p. 5 and your reply treat M3−M2 as a change of scale. If M2 uses a global baseline, M2 ≈ M3 (54.35 vs 54.25) does not single out the within-group baseline, as the abstract does. Which is meant? (c) Table 5 places the ladder behind the abstract's +1.72 and +15.20 under the submitted code S1, and only M1/M3 were rerun. The Table 9 caption could say so and the abstract quote the released-code +14.18.
-
W2/Q3 (scope of the claims). Thank you for agreeing on scope and for the all-local bound, new abstract ending and narrowed conclusion. The abstract's five-seed 55.33/0.45 and Table 2's S5 block have no all-local reference, though the control's 53.31% SR exceeds every non-privileged baseline there. Could both show it? The p. 2 heading now says "stated before the experiments", while your reply says the criterion was induced from the failures. My Q3 did say "before the experiments", but I meant before a test that could refute it. A heading matching your reply would settle this.
-
Q2 (COMA). A.9 infers a zero advantage from equal per-seed means of Q and b, but since b averages Q over the agent's own action under the policy, Q − b has zero mean in expectation for any Q. The "numerically zero" sentence could rest on the behavioral evidence.
Minor: a table of λ_v, λ_c, λ_g and the PPO settings (Eq. 5 also lacks the 1/5 of Algorithm 1, line 9); a citation for difference rewards in A.9; and a budget and code state per caption, since the full method's COV is 0.45 (Table 2), 2.33 (Table 7), 2.78 (Table 12) and 2.37 (Table 13).
Thank you again.
Response to Reviewer 1ETU
We thank the reviewer for the constructive report and for recognizing the mechanism analysis and the honest reporting of negative results. The four questions identify the right pressure points: extra information, comparison with existing counterfactual methods, scope, and implementation details. We answer each in turn, and we agree with the scope criticism.
Before the point-by-point reply, we summarize the common question across the three reviews: whether the gain comes from extra simulator access rather than from the mechanism. We ran the comparison at fixed information. M1 and M3 draw and score the same G = 8 candidates per agent per step (160 evaluations per step) and receive the same VFR supervision and the same validity and certificate terms in the score; M3 additionally uses those scores as a within-group baseline. In the released code, with all four fixes active, over five seeds × 1,000 episodes (last 500), M1 reaches 38.64 ± 1.95% SR and M3 reaches 52.82 ± 0.77%, a paired difference of +14.18 points (p = 7 × 10⁻⁵, 5/5 seeds positive). M1 is only +1.72 points above M0, which uses no candidate scores (p = 0.44).
Three code states are kept separate. S1 is the submitted artifact (fixed bandwidth denominator, EMA peers, what-if certificate target, COV over all users). S2 is the bandwidth-only retrain (proportional bandwidth, exact bandwidth-demand peers, EMA CPU, what-if target, COV over all users); only this fix changes the executed physics, so only this one was fully retrained. S3 is the released code used for every new result: proportional bandwidth, exact CPU and bandwidth peer context from the registered candidate 0, VFR target from the realized step(), and active-user COV. The S3 headline pair is the check that the remaining training-signal fixes do not erase the result.
W1/Q1 — the gain is not disentangled from extra information
The fixed-information ladder separates them. M1 receives the same 160 queries, the same what-if rewards and certificates, and the same VFR supervision as M3, including the validity term −λ_c · viol(a_g, ĉ_i); it differs only in that the scores are not used as a group baseline. M1 is statistically indistinguishable from M0, which uses 20 queries and no scores (+1.72, p = 0.44). Adding the group baseline at the same 160 queries gives +14.18 points (p = 7 × 10⁻⁵, 5/5). Permuting the scores within the group returns to the M0 level (−2.66 vs. M0, p = 0.53), and a global scale instead of the per-group scale changes nothing (−0.10, p = 0.93). The validity reward is therefore present in the control arm; what the control lacks is the within-group comparison.
A baseline can obtain the same information by paying the same 8× simulator cost. What it cannot obtain for free is the comparison of its own candidate outcomes within the group. The M1 arm is exactly the "same extra information" baseline requested in this question.