- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Multi-Granularity Modulated (MGM) credit assignment for long-horizon LLM agents. MGM combines token-level Top-K entropy filtering to construct step-level uncertainty estimates, step-level temporal modulation to differentiate early exploration from late-stage execution, and task-level weighting to prioritize challenging queries. Experiments on ALFWorld, WebShop, and Search-Augmented QA demonstrate improvements over GRPO, DAPO, and other relevant baselines.
Critical strengths
- The paper addresses an important credit assignment problem in long-horizon agent training. MGM can be integrated into existing group-based RL methods without requiring a learned critic.
- The experiment improvements over the baselines are substantial, particularly on ALFWorld.
- The paper provides useful empirical analysis. The Top-K sensitivity experiments, and training dynamics provide insights into the behavior of the proposed method's mechanisms.
Critical weaknesses
- The paper does not have enough ablations of the proposed method. Table 10 includes only individual components and the full model, omitting pairwise combinations. Also, the ablation is only conducted using ALFWorld and Qwen2.5-1.5B-Instruct.
- The experiment is only conducted on Qwen 2.5 models. There should be more experiment validation on newer models and other model families beyond Qwen.
- The proposed method seems to be applicable to search or interactive agents. Can the authors also validate the proposed method on tool-augmented math reasoning and math reasoning without tools since token-level entropy filtering and task-level difficulty weighting do not inherently require multi-turn interaction.
Questions for authors
Please see weaknesses
3 comments
Q2 (Weakness 2): The experiment is only conducted on Qwen 2.5 models. There should be more experiment validation on newer models and other model families beyond Qwen.
A2: Thank you for the suggestion. Our model selection was guided by the goal of evaluating MGM's incremental contribution as a post-training method. We selected Qwen2.5 at multiple sizes and Llama-3.2-3B as established backbones from different model families with conventional full-attention Transformer architectures. Comparing MGM with the corresponding optimization baselines on each backbone allows us to assess its contribution, while the two model families help examine whether the benefits extend beyond a single family.
The submitted paper already includes a cross-family evaluation on Llama-3.2-3B-Instruct in Appendix E.6, Table 11. On ALFWorld, MGM improves its success rate from 58.6% to 61.7%, a gain of 3.1 percentage points over GRPO. This complements the Qwen2.5-1.5B and 7B evaluations already reported in the main text.
We agree that evaluation on newer backbones is valuable. Qwen3.5 introduces a hybrid linear/full-attention architecture and a different upstream training recipe, including large-scale reinforcement learning [2]. We therefore test whether MGM also improves performance on Qwen3.5-2B.
To extend the evaluation to a newer backbone, we completed a controlled comparison on ALFWorld with Qwen3.5-2B, evaluating GRPO, GRPO + EMPG, and GRPO + MGM (Ours). We adapted the existing verl-agent codebase [1] to support Qwen3.5-2B and used the same experimental protocol for the three methods. Each method uses a single random seed on two NVIDIA A100 GPUs.
Table R2. Completed ALFWorld evaluation with Qwen3.5-2B. All metrics are success rates (%), covering the six task categories and the overall evaluation (All). Results are reported for one seed per method.
| Method | Pick | Look | Clean | Heat | Cool | Pick2 | All |
|---|---|---|---|---|---|---|---|
| GRPO | 83.9 | 83.3 | 68.4 | 55.0 | 76.9 | 85.0 | 75.8 |
| GRPO + EMPG | 85.7 | 81.8 | 100.0 | 54.5 | 75.0 | 57.9 | 80.5 |
| GRPO + MGM (Ours) | 84.6 | 84.6 | 93.1 | 100.0 | 81.0 | 58.8 | 88.3 |
The Llama evaluation examines transfer across model families, while the Qwen3.5-2B comparison tests performance on a newer backbone. The overall success rates are 75.8% for GRPO, 80.5% for EMPG, and 88.3% for MGM. MGM improves over GRPO by 12.5 percentage points and over EMPG by 7.8 points, with gains on five of the six task categories relative to GRPO; Pick2 is lower for MGM in this run.
References
[1] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-Group Policy Optimization for LLM Agent Training. arXiv:2505.10978, 2025.
[2] Qwen Team. Qwen3.5-2B: Official Model Card.
Q3 (Weakness 3): The proposed method seems to be applicable to search or interactive agents. Can the authors also validate the proposed method on tool-augmented math reasoning and math reasoning without tools since token-level entropy filtering and task-level difficulty weighting do not inherently require multi-turn interaction.
A3: Thank you for this interesting suggestion. Our Search-Augmented QA experiments already evaluate tool-augmented reasoning: agents issue search queries, retrieve external information, and integrate the evidence to answer single- and multi-hop questions. The setup allows at most four interaction turns, as specified in Appendix E.1/Table 5. We included this task to examine MGM in shorter-horizon tool-augmented reasoning, complementing the longer interactive trajectories in ALFWorld and WebShop. Table 2 reports an average score of 45.0 versus GiGPO's 42.1 in this setting.
We agree that MGM's components also have natural applications to mathematical reasoning. In SimpleTIR, for example, the model iteratively reasons, generates code, and uses execution feedback to continue solving a problem. Each model-generated turn maps naturally to a step in MGM. We would retain SimpleTIR's void-turn trajectory filtering and apply MGM to the retained trajectories, with the following expected benefits:
- Token-level modulation: Aggregating high-entropy positions makes the turn-level uncertainty estimate less diluted by highly predictable continuations in reasoning text and code. This should support better-calibrated updates when the model chooses a solution strategy, constructs a computation, or revises its reasoning after tool feedback.
- Step-index modulation: Later turns receive greater relative weight, strengthening learning from feedback-informed corrections, verification, and final-answer construction, while retaining a learning signal for early problem decomposition and exploratory tool use. On unsuccessful trajectories, the same weighting reduces the relative penalty on early exploration.
- Task-difficulty modulation: Once the model has acquired basic competence, problems with lower group-average correctness receive greater training weight. This should sustain learning on challenging mathematical problems as easier ones become mastered.
Together, these components are expected to improve the allocation of terminal correctness rewards across reasoning turns and across problems, supporting more effective learning of multi-turn mathematical solution strategies. For mathematics without tools, token-level uncertainty estimation and task-difficulty weighting remain directly applicable; a single-response solution can use a temporal factor of one. We will add end-to-end experiments in a future version, comparing SimpleTIR with SimpleTIR + MGM, examining the component contributions, and evaluating the token- and task-level components in a matched tool-free mathematical-reasoning setting.
We thank the reviewer for the concrete scope questions.
A1: Thank you for the suggestion. The individual-component study in Table 10 complements the incremental ablation in the main text. In the submitted paper, Table 3 reports the DAPO baseline, T, T+S, T+D, and T+S+D, while Appendix E.5/Table 10 additionally reports S-only and D-only. Together, these tables covered seven of the eight configurations, with S+D as the only missing combination. Section 4.4 also explicitly directed readers to Appendix E.5.
We have now completed the remaining S+D experiment, which achieves 90.6% overall success, and consolidated all eight configurations into a single table in Appendix E.5, Table 10. This brings the baseline, individual components, pairwise combinations, and full method together, making the complete comparison directly accessible in one place. The consolidated study uses the same ALFWorld/Qwen2.5-1.5B setting to examine the components under a common task and backbone.
Table R1. Complete component ablation on ALFWorld with Qwen2.5-1.5B-Instruct. T denotes token-level modulation, S step-index modulation, and D task-difficulty modulation. Values are success rates (%). The table combines the submitted ablation results with the newly completed S+D experiment.