- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CovGPO addresses zero-reward groups whose outcome-relative advantages vanish. It ranks failed trajectories by first-visit observation coverage and conditionally substitutes normalized coverage advantages. The method requires no additional learned components or rollouts. Experiments augment GRPO, GiGPO and HGPO on ALFWorld and WebShop, with an exploratory Sokoban extension.
Critical strengths
-
Low-cost integration. CovGPO reuses collected trajectories without extra models or environment interactions, making it a practical augmentation of existing optimizers.
-
Relevant experimental controls. Comparisons with always-on coverage and substituted signals support the usefulness of conditional, trajectory-aligned feedback in the tested settings.
Critical weaknesses
-
The contribution is incremental, and the coverage signal does not adequately capture semantic redundancy. The main modification is a conditional replacement of advantages using a trajectory-level exploration statistic. Appendix C.2 specifies exact observation-string matching rather than semantic equivalence. Equivalent actions can therefore contribute to coverage when their outputs differ superficially, while hidden state changes may go undetected. The authors acknowledge this limitation, but do not directly test representation sensitivity. This weakens the case for a broadly reliable credit-assignment improvement.
-
Performance gains are uneven, and broader effectiveness remains insufficiently established. Although some improvements over GiGPO are meaningful, aggregate gains over GRPO and the stronger 7B HGPO baseline are modest. Table 1 also shows substantial subtask variation, including gains on Look and Cool but regressions on Clean and Heat with the 1.5B GRPO base. The category breakdowns and diagnostics do not adequately explain these regressions. Text experiments use only 1.5B and 7B models on two benchmarks, while Sokoban changes the signal. Consistent benefits with larger models or more diverse tasks therefore remain uncertain.
-
The proofs provide limited support for task-performance improvement. Appendix B.4 assumes that greater coverage implies better continuation value rather than establishing this relationship. B.6 proves a gradient identity for an unnormalized estimator of coverage conditioned on failure, not for the deployed gated and normalized update. B.7 further shows that fallback credit can suppress a key forward action. These results characterize the mechanism but do not establish a theoretical link between coverage-based updates and escaping the credit trap.
-
Presentation suggestion. The main idea is understandable, but the frequent colon-led lists make the exposition fragmented. Related work should be integrated into the main paper.
Questions for authors
see weaknesses above
11 comments
Response to Reviewer qJMP – Part 2
Q2: Controlled Protocol (Appendix C.2), Direct Representation-Sensitivity Test, and Fail-Safe Gating
A: Regarding your concern in W1 that "Appendix C.2 specifies exact observation-string matching rather than semantic equivalence; equivalent actions can contribute to coverage when outputs differ superficially, while hidden state changes may go undetected, and the authors do not directly test representation sensitivity," our 4-benchmark string-similarity sweep directly quantifies representation sensitivity (showing that relaxing exact matching incurs a 5× to 27× state-contamination penalty) while our gate safely neutralizes superficial output differences and hidden state changes.
-
Controlled protocol & description of Table R6: First, Appendix C.2 adopts exact observation matching () to strictly follow GiGPO (Feng et al., 2025, arXiv:2505.10978) in
verl-agent, ensuring CovGPO and step-level baselines (GiGPO, HGPO) operate on identical anchor groups without confounding state abstraction with credit assignment. Second, to directly test representation sensitivity, we replayed stored rollouts (256 trajectories per setting on ALFWorld/ScienceWorld/WebShop; 320 on SearchQA) through GiGPO's approximate string-similarity grouping across , measuring multi-member grouping rate, non-zero advantage Signal rate (), and false state-merging Contam rate (% of execution steps). -
Summary of findings from Table R6 & fail-safe analysis:
- Signal-noise-ratio degradation under relaxed string matching: Lowering from increases non-zero advantage steps by only +0.0 to 5.1 pp, while falsely merging genuinely distinct states in 20.5% to 54.9% of steps (a 5× to 27× noise-to-signal penalty; e.g., conflating distinct rooms or objects sharing surface templates).
- Open-ended retrieval (
SearchQA) stability: Exact matching () groups 76.2% (1.5B) / 76.9% (7B) of steps and remains nearly invariant across (0.0% to 0.3% contamination). - Fail-safe gating under superficial output differences and hidden state changes: In edge cases where superficial output differences mark every step novel () or undetected hidden state changes leave observations unchanged across the group (), within-group dispersion vanishes () and our variance gate automatically withholds the fallback, reducing CovGPO exactly to the base learner (Proposition 2) instead of injecting corrupted advantages.
Revision: We will include Table R6 and the superficial/hidden-state fail-safe proof discussion in Appendix C.2 and Section 4.3.
Q3: Subtask Regressions Under 1.5B GRPO vs. Simultaneous Gains Under Step-Level GiGPO and HGPO
A: Regarding your observation in W2 that "Table 1 shows substantial subtask variation, including gains on Look and Cool but regressions on Clean and Heat with the 1.5B GRPO base, and the category breakdowns do not adequately explain these regressions," 1.5B GRPO's drops on Clean and Heat arise from broadcasting a single coarse trajectory-wide scalar across 20 to 50 steps onto shared low-level primitives under compact 1.5B capacity, whereas step-level grouping in GiGPO/HGPO protects mastered states and improves all 6 subtasks simultaneously.
- Coarse trajectory-level broadcasting and shared-primitive interference in 1.5B GRPO: All six ALFWorld subtasks share common low-level interaction primitives (
goto,take,open,put), while the three hardest subtasks Pick2 (34%), Cool (26%), and Look (18%), account for 78% of all-failed groups (Appendix Table 17). Vanilla GRPO assigns zero advantage () to these failed groups, leaving Look (53.7%) and Cool (59.7%) under-trained while concentrating updates on easier subtasks* (Clean and Heat). Recovering gradients on all-failed groups enables 1.5B GRPO+CovGPO to substantially boost Look (+22.6 pp) and Cool (+23.5 pp) and lift overall success (). However, because GRPO broadcasts a single trajectory-wide scalar advantage uniformly across all 20 to 50 steps, a high-coverage failed trajectory that explores new rooms but later wanders receives positive credit on every step, whereas a low-coverage failed trajectory that executes valid early navigation/pickup primitives before stalling is penalized across its entire prefix. Under compact 1.5B capacity, this coarse trajectory-wide update introduces gradient interference on shared primitives and trades off part of Clean and Heat (where 1.5B GRPO already shows high seed variance of ± 6.8 to 9.2).
Response to Reviewer qJMP – Part 3
Table R7: ALFWorld 1.5B Subtask Breakdown vs. All-Failed Group Composition (Main Table 1 & Appendix Table 17).
| Subtask | Share of All-Failed Groups | 1.5B GRPO +CovGPO | 1.5B GiGPO +CovGPO | 1.5B HGPO +CovGPO |
|---|---|---|---|---|
| Pick2 (Hard) | 34% | (-2.7) | (+14.5) | (+5.5) |
| Cool (Hard) | 26% | (+23.5) | (+10.8) | (+4.7) |
| Look (Hard) | 18% | (+22.6) | (+7.0) | (+5.9) |
| Clean | 12% | (-9.7) | (+3.6) | (+3.3) |
| Heat | 6% | (-8.6) | (+1.5) | (-0.3) |
| Pick (Saturated) | 4% | (-1.2) | (+0.5) | (+2.2) |
| Overall | 100% | (+1.7) | (+5.5) | (+2.9) |
- Summary of findings from Table R7: Once paired with step-level credit assignment, 1.5B GiGPO+CovGPO improves all 6 subtasks simultaneously (, led by +14.5 pp on Pick2 and +10.8 pp on Cool) without any regression on Clean (+3.6 pp) or Heat (+1.5 pp), 1.5B HGPO+CovGPO reaches 95.7% overall, and at larger model capacity 7B GRPO+CovGPO likewise boosts Heat from .
Revision: We will add this mechanistic diagnosis of trajectory-wide broadcasting interference vs. step-level local gating to Section 4.1.
Q4: Non-Interference on Near-Saturated 7B HGPO and Broader Task/Model Validation
A: Regarding your concern in W2 that "aggregate gains over the stronger 7B HGPO baseline are modest, text experiments use only 1.5B and 7B models on two benchmarks, so consistent benefits with diverse tasks and models remain uncertain," on near-saturated 7B HGPO (95.4%), Proposition 2 guarantees non-interfering disengagement, while expanded runs across harder text benchmarks (SearchQA, ScienceWorld) and newer model families (Qwen3.5-2B/4B).
Table R8: Expanded Evaluation Across Harder Benchmarks and Newer Model Generations.
| Method | NQ† | TriviaQA★ | PopQA★ | HotpotQA† | 2Wiki★ | MuSiQue★ | Bamboogle★ | Avg. | ScienceWorld SR | ScienceWorld Score |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen2.5-3B-Instruct | ||||||||||
| GRPO | 45.24 ± 0.29 | 60.64 ± 0.06 | 45.37 ± 0.09 | 36.37 ± 0.18 | 35.21 ± 0.17 | 12.26 ± 0.32 | 26.67 ± 0.92 | 37.40 ± 0.12 | - | - |
| GRPO+CovGPO | 43.46 ± 0.28 | 60.08 ± 0.04 | 44.29 ± 0.26 | 39.45 ± 0.25 | 39.46 ± 0.14 | 14.98 ± 0.65 | 29.87 ± 2.01 | 38.80 ± 0.29 | - | - |
| GiGPO | 44.9 ± 0.3 | 60.4 ± 0.0 | 46.7 ± 0.1 | 36.3 ± 0.2 | 35.0 ± 0.4 | 12.0 ± 0.3 | 63.0 ± 0.5 | 42.6 ± 0.0 | - | - |
| GiGPO+CovGPO | 42.4 ± 0.6 | 59.8 ± 0.1 | 45.6 ± 0.3 | 39.3 ± 0.3 | 39.6 ± 0.2 | 14.7 ± 0.7 | 64.7 ± 1.0 | 43.7 ± 0.2 | - | - |
| Qwen2.5-7B-Instruct | ||||||||||
| GRPO | 41.6 ± 2.3 | 60.7 ± 0.9 | 43.5 ± 1.4 | 39.6 ± 0.0 | 37.7 ± 2.6 | 15.2 ± 0.5 | 49.9 ± 25.2 | 41.2 ± 2.9 | 21.6 ± 2.3 | 29.3 ± 2.6 |
| GRPO+CovGPO | 46.0 ± 1.3 | 63.2 ± 0.4 | 45.7 ± 0.7 | 40.4 ± 0.1 | 42.2 ± 0.1 | 18.1 ± 0.9 | 69.5 ± 1.5 | 46.5 ± 0.3 | 22.9 ± 2.3 | 30.1 ± 4.0 |
| GiGPO | 40.2 ± 0.4 | 61.3 ± 0.1 | 41.8 ± 1.0 | 39.5 ± 0.2 | 37.5 ± 2.2 | 15.7 ± 0.1 | 67.9 ± 0.3 | 43.4 ± 0.3 | 22.9 ± 2.3 | 29.8 ± 4.3 |
| GiGPO+CovGPO | 45.4 ± 0.3 | 63.7 ± 0.6 | 46.4 ± 0.2 | 42.7 ± 0.0 | 41.1 ± 1.1 | 18.9 ± 0.1 | 72.2 ± 1.7 | 47.2 ± 0.1 | 25.3 ± 0.9 | 36.3 ± 1.7 |
Response to Reviewer qJMP – Part 1
We sincerely thank you for the detailed and constructive feedback. We address each weakness and suggestion point by point below.
Q1: Value and Core Novelty of Lightweight Signal Extraction from All-Failed Rollouts
A: Regarding your concern in W1 that "the contribution is incremental... the main modification is a conditional replacement of advantages using a trajectory-level exploration statistic," we respectfully view uncovering an insightful perspective through a lightweight (O(1)), non-interfering mechanism as a deliberate strength: our core novelty lies in extracting credible, structured signals from degenerate all-failed trajectories so rollout compute is fully utilized rather than discarded.
Revision: We will sharpen this core contribution framing and broader uniform-outcome signal-extraction perspective in Section 1 and Section 3.2.
Table R6: Offline String-Similarity Threshold (τsim) Representation-Sensitivity Sweep Across 4 Benchmarks with Qwen2.5.