acceptodds
Under review as a conference paper at ICLR 2027

Focus on the Critical: Multi-Granularity Modulated Credit Assignment for Long-Horizon LLM Agents

Abstract

Efficient credit assignment is critical for reinforcement learning (RL) of large language model (LLM) agents in long-horizon interactive environments, where supervision is typically sparse and delayed. However, group-based policy optimization methods often apply a trajectory-level signal uniformly across steps and tokens, which can obscure a small number of pivotal decisions that determine task success. To address this limitation, we propose a *Multi-Granularity Modulated* (MGM) credit assignment framework. It refines the learning signal by integrating token-level Top- entropy filtering to isolate critical decisions, step-level temporal modulation to differentiate early exploration from late-stage execution, and task-level weighting to prioritize challenging queries. Through this unified hierarchical design, MGM effectively highlights pivotal actions, enabling more accurate and efficient policy optimization. Extensive experiments on ALFWorld, WebShop, and Search-Augmented QA demonstrate that our method consistently outperforms state-of-the-art baselines, up to 25.0% absolute improvement based on GRPO and 14.5% based on DAPO. Moreover, we observe a robust unimodal sensitivity to the Top- ratio, with best performance typically achieved when retaining roughly –% of tokens for entropy estimation, suggesting that excluding a small fraction of low-entropy tokens can improve the signal-to-noise ratio in long-horizon credit assignment.

Primary area:
reinforcement learning
Keywords:
LLM Agents, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dz3wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Multi-Granularity Modulated (MGM) credit assignment for long-horizon LLM agents. MGM combines token-level Top-K entropy filtering to construct step-level uncertainty estimates, step-level temporal modulation to differentiate early exploration from late-stage execution, and task-level weighting to prioritize challenging queries. Experiments on ALFWorld, WebShop, and Search-Augmented QA demonstrate improvements over GRPO, DAPO, and other relevant baselines.

Critical strengths
  1. The paper addresses an important credit assignment problem in long-horizon agent training. MGM can be integrated into existing group-based RL methods without requiring a learned critic.
  2. The experiment improvements over the baselines are substantial, particularly on ALFWorld.
  3. The paper provides useful empirical analysis. The Top-K sensitivity experiments, and training dynamics provide insights into the behavior of the proposed method's mechanisms.
Critical weaknesses
  1. The paper does not have enough ablations of the proposed method. Table 10 includes only individual components and the full model, omitting pairwise combinations. Also, the ablation is only conducted using ALFWorld and Qwen2.5-1.5B-Instruct.
  2. The experiment is only conducted on Qwen 2.5 models. There should be more experiment validation on newer models and other model families beyond Qwen.
  3. The proposed method seems to be applicable to search or interactive agents. Can the authors also validate the proposed method on tool-augmented math reasoning and math reasoning without tools since token-level entropy filtering and task-level difficulty weighting do not inherently require multi-turn interaction.
Questions for authors

Please see weaknesses

3 comments

We thank the reviewer for the concrete scope questions.

Q1 (Weakness 1): The paper does not have enough ablations of the proposed method. Table 10 includes only individual components and the full model, omitting pairwise combinations. Also, the ablation is only conducted using ALFWorld and Qwen2.5-1.5B-Instruct.

A1: Thank you for the suggestion. The individual-component study in Table 10 complements the incremental ablation in the main text. In the submitted paper, Table 3 reports the DAPO baseline, T, T+S, T+D, and T+S+D, while Appendix E.5/Table 10 additionally reports S-only and D-only. Together, these tables covered seven of the eight configurations, with S+D as the only missing combination. Section 4.4 also explicitly directed readers to Appendix E.5.

We have now completed the remaining S+D experiment, which achieves 90.6% overall success, and consolidated all eight configurations into a single table in Appendix E.5, Table 10. This brings the baseline, individual components, pairwise combinations, and full method together, making the complete comparison directly accessible in one place. The consolidated study uses the same ALFWorld/Qwen2.5-1.5B setting to examine the components under a common task and backbone.

Table R1. Complete component ablation on ALFWorld with Qwen2.5-1.5B-Instruct. T denotes token-level modulation, S step-index modulation, and D task-difficulty modulation. Values are success rates (%). The table combines the submitted ablation results with the newly completed S+D experiment.

ConfigurationPickLookCleanHeatCoolPick2AllSource
DAPO baseline88.161.482.590.183.969.580.8Tables 3 and 10
T only96.478.6100.082.488.980.089.1Tables 3 and 10
S only100.094.1100.085.777.882.691.4Appendix E.5, Table 10
D only92.166.795.5100.081.884.288.3Appendix E.5, Table 10
T+S100.096.9100.0100.085.786.793.0Section 4.4, Table 3
T+D100.083.394.7100.095.582.193.0Section 4.4, Table 3
S+D94.387.596.3100.090.073.990.6New experiment
T+S+D100.0100.0100.090.996.280.095.3Tables 3 and 10

Q2 (Weakness 2): The experiment is only conducted on Qwen 2.5 models. There should be more experiment validation on newer models and other model families beyond Qwen.

A2: Thank you for the suggestion. Our model selection was guided by the goal of evaluating MGM's incremental contribution as a post-training method. We selected Qwen2.5 at multiple sizes and Llama-3.2-3B as established backbones from different model families with conventional full-attention Transformer architectures. Comparing MGM with the corresponding optimization baselines on each backbone allows us to assess its contribution, while the two model families help examine whether the benefits extend beyond a single family.

The submitted paper already includes a cross-family evaluation on Llama-3.2-3B-Instruct in Appendix E.6, Table 11. On ALFWorld, MGM improves its success rate from 58.6% to 61.7%, a gain of 3.1 percentage points over GRPO. This complements the Qwen2.5-1.5B and 7B evaluations already reported in the main text.

We agree that evaluation on newer backbones is valuable. Qwen3.5 introduces a hybrid linear/full-attention architecture and a different upstream training recipe, including large-scale reinforcement learning [2]. We therefore test whether MGM also improves performance on Qwen3.5-2B.

To extend the evaluation to a newer backbone, we completed a controlled comparison on ALFWorld with Qwen3.5-2B, evaluating GRPO, GRPO + EMPG, and GRPO + MGM (Ours). We adapted the existing verl-agent codebase [1] to support Qwen3.5-2B and used the same experimental protocol for the three methods. Each method uses a single random seed on two NVIDIA A100 GPUs.

Table R2. Completed ALFWorld evaluation with Qwen3.5-2B. All metrics are success rates (%), covering the six task categories and the overall evaluation (All). Results are reported for one seed per method.

MethodPickLookCleanHeatCoolPick2All
GRPO83.983.368.455.076.985.075.8
GRPO + EMPG85.781.8100.054.575.057.980.5
GRPO + MGM (Ours)84.684.693.1100.081.058.888.3

The Llama evaluation examines transfer across model families, while the Qwen3.5-2B comparison tests performance on a newer backbone. The overall success rates are 75.8% for GRPO, 80.5% for EMPG, and 88.3% for MGM. MGM improves over GRPO by 12.5 percentage points and over EMPG by 7.8 points, with gains on five of the six task categories relative to GRPO; Pick2 is lower for MGM in this run.

References

[1] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-Group Policy Optimization for LLM Agent Training. arXiv:2505.10978, 2025.

[2] Qwen Team. Qwen3.5-2B: Official Model Card.

Q3 (Weakness 3): The proposed method seems to be applicable to search or interactive agents. Can the authors also validate the proposed method on tool-augmented math reasoning and math reasoning without tools since token-level entropy filtering and task-level difficulty weighting do not inherently require multi-turn interaction.

A3: Thank you for this interesting suggestion. Our Search-Augmented QA experiments already evaluate tool-augmented reasoning: agents issue search queries, retrieve external information, and integrate the evidence to answer single- and multi-hop questions. The setup allows at most four interaction turns, as specified in Appendix E.1/Table 5. We included this task to examine MGM in shorter-horizon tool-augmented reasoning, complementing the longer interactive trajectories in ALFWorld and WebShop. Table 2 reports an average score of 45.0 versus GiGPO's 42.1 in this setting.

We agree that MGM's components also have natural applications to mathematical reasoning. In SimpleTIR, for example, the model iteratively reasons, generates code, and uses execution feedback to continue solving a problem. Each model-generated turn maps naturally to a step in MGM. We would retain SimpleTIR's void-turn trajectory filtering and apply MGM to the retained trajectories, with the following expected benefits:

  • Token-level modulation: Aggregating high-entropy positions makes the turn-level uncertainty estimate less diluted by highly predictable continuations in reasoning text and code. This should support better-calibrated updates when the model chooses a solution strategy, constructs a computation, or revises its reasoning after tool feedback.
  • Step-index modulation: Later turns receive greater relative weight, strengthening learning from feedback-informed corrections, verification, and final-answer construction, while retaining a learning signal for early problem decomposition and exploratory tool use. On unsuccessful trajectories, the same weighting reduces the relative penalty on early exploration.
  • Task-difficulty modulation: Once the model has acquired basic competence, problems with lower group-average correctness receive greater training weight. This should sustain learning on challenging mathematical problems as easier ones become mastered.

Together, these components are expected to improve the allocation of terminal correctness rewards across reasoning turns and across problems, supporting more effective learning of multi-turn mathematical solution strategies. For mathematics without tools, token-level uncertainty estimation and task-difficulty weighting remain directly applicable; a single-response solution can use a temporal factor of one. We will add end-to-end experiments in a future version, comparing SimpleTIR with SimpleTIR + MGM, examining the component contributions, and evaluating the token- and task-level components in a matched tool-free mathematical-reasoning setting.

Reviewer_jpUhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MGM modulates step-level advantages in multi-turn agent RL using entropy estimated from high-entropy tokens, future-step clarity, and position- and difficulty-dependent factors. Experiments on interactive environments and question answering report substantial gains over several RL baselines, with component ablations and an additional model-family evaluation.

Critical strengths
  1. The empirical improvements are substantial across multiple environments. On ALFWorld with the 1.5B model, the reported DAPO result improves from 80.8 to 95.3 with MGM.
  2. The appendix includes factorial ablations and a Llama evaluation. These support the usefulness of the components and provide evidence beyond a single backbone.
Critical weaknesses
  1. The theoretical claims exceed what the stated bounds establish. Lemma 3.1 allows K=1 and therefore does not by itself establish a beneficial nontrivial truncation or justify a shared 80-90% choice. Theorem 3.2 tightens an upper bound on modulation error, but a tighter upper bound does not prove that the actual error is smaller.
  2. The interpretation as semantic denoising remains insufficiently established. Selecting high-entropy tokens necessarily increases the average entropy; the performance gains and component ablations do not by themselves show that the removed tokens are specifically semantic noise. This weakens a central explanation of why the method works.
Questions for authors
  1. Can the authors revise the theoretical conclusion to distinguish an error-bound improvement from an actual error ordering, and state what is guaranteed for a nontrivial shared K?
  2. What evidence connects the selected tokens to semantic relevance, beyond entropy and downstream scores? A controlled comparison or narrower interpretation would materially affect my assessment.
3 comments

We thank the reviewer for the careful reading and for identifying where the theoretical and semantic claims need clarification.

Q1 (Weakness 1): The theoretical claims exceed what the stated bounds establish. Lemma 3.1 allows K=1 and therefore does not by itself establish a beneficial nontrivial truncation or justify a shared 80-90% choice. Theorem 3.2 tightens an upper bound on modulation error, but a tighter upper bound does not prove that the actual error is smaller.

A1: Thank you for pointing out the distinction between an error bound and a realized-error ordering. Theorem 3.2 provides a quantitative guarantee on how uncertainty-estimation errors affect credit weights after mean normalization. We will state the operational meaning of this guarantee more clearly and distinguish it from the stronger comparison in the current explanatory text.

Estimator accuracy versus modulation stability. For a retention choice covered by its existence statement, Lemma 3.1 establishes an actual comparison of errors relative to the defined semantic oracle: ∣H^tK−Ut∣≤∣H^tall−Ut∣|\widehat H_t^K-U_t|\le |\widehat H_t^{\mathrm{all}}-U_t|. Theorem 3.2 then propagates estimation error through the mean-normalized multiplier. Under its Lipschitz assumptions, with the same positive priors Bt=S(t)D(x)B_t=S(t)D(x) held fixed,

\le 2L\|\widehat H-U\|_\infty.$$ **What the bound guarantees.** Writing $\varepsilon=\|\widehat H-U\|_\infty$, the theorem implies, for every step, $$e^{-2L\varepsilon}\le \frac{M_t(\widehat H)}{M_t(U)} \le e^{2L\varepsilon}.$$ This controls the multiplicative distortion of each credit weight relative to the same-prior oracle. Reducing the estimator error contracts this guaranteed range of weight distortion, excluding deviations that the larger bound cannot rule out. This matters because batch normalization couples every step's weight to every other step: the theorem shows that this coupling still permits uniform error control, with no additional factor proportional to the number of steps or the ratio of the largest to smallest positive prior. It therefore gives a concrete reason to improve the uncertainty estimator used inside MGM. Under the same scaling function and priors, an estimator approaching the oracle also yields credit weights approaching the oracle weights. This is an approximation guarantee; downstream benefits are evaluated experimentally. The sentence immediately following Theorem 3.2 currently states that Top-K produces an allocation "closer to the oracle semantic allocation than all-token entropy," and the appendix proof ends with the same comparison. We will revise these two sentences to describe the tighter distortion bound above. We agree that this bound does not order the two realized modulation errors: both methods may lie well inside their respective bounds. This qualification preserves the quantitative guarantee while correcting the stronger interpretation. **A sufficient condition for a nontrivial shared K.** To address the reviewer's question about a shared retention ratio below one, we will supplement the existence result with an explicit sufficient condition linking K to the oracle estimation gap and the token-entropy range. Suppose that, throughout a batch, $$U_t-\widehat H_t^{\mathrm{all}}\ge\Delta>0, \qquad \max_j h_{t,j}-\min_j h_{t,j}\le R_H.$$ For a shared retention ratio $K$ and $r_t=\lceil Km_t\rceil$, Top-K averaging satisfies $$0\le\widehat H_t^K-\widehat H_t^{\mathrm{all}} \le(1-K)R_H.$$ Consequently, $K\ge 1-\Delta/R_H$ guarantees $\widehat H_t^{\mathrm{all}}\le\widehat H_t^K\le U_t$ at every step. The oracle estimation error is strictly smaller wherever $r_t<m_t$. If truncation is required at every step, a sufficient shared interval is $$1-\frac{\Delta}{R_H}\le K\le 1-\frac{1}{m_{\min}}, \qquad m_{\min}=\min_t m_t,$$ provided this interval is nonempty. This identifies explicit additional conditions for a nontrivial shared K without assuming in advance that Top-K is more accurate. This is a proposed additional sufficient condition, not a consequence of the original assumptions alone or a claim that the experimental batches satisfy it. Since $\Delta$ depends on the latent oracle, it is not a directly observable tuning rule. The reported 80–90% range remains an empirical sensitivity finding; neither this condition nor Theorem 3.2 establishes a universal numerical K or an ordering of downstream performance.

Q2 (Weakness 2): The interpretation as semantic denoising remains insufficiently established. Selecting high-entropy tokens necessarily increases the average entropy; the performance gains and component ablations do not by themselves show that the removed tokens are specifically semantic noise. This weakens a central explanation of why the method works.

A2: Thank you for asking what the filtering retains and excludes. We added a five-case token-level study on ALFWorld with Qwen2.5-1.5B-Instruct in Appendix F.1, "Token-Level Case Study," and a retention-matched random-mask analysis in Appendix F.2, "Retention-Matched Random Mask Analysis." Each case shows eligible response tokens in their original order, with entropy color-coded and Top-80% omissions underlined; the mask is computed over the full response. These examples make the filtering behavior inspectable beyond aggregate entropy or task scores.

The intended target is the step's semantic decision pressure conditioned on the prefix. A token can be semantically essential while its realization is predictable from preceding text. Our modulation reduces this dilution in the step-uncertainty estimate. Token category is therefore not our definition of relevance: object names need not always be retained, nor punctuation and function words always removed. The five cases illustrate this distinction:

CaseObservation in the recorded responseInterpretation
1: Subword continuationIn the first occurrence of countertop, countert is retained (entropy 3.086 nats), while the continuation op is omitted (entropy 0.000183 nats).A meaningful word can contain a highly predictable continuation; filtering is applied to generation positions, not to the semantic importance of the whole word.
2: The same lexical token in different contextsThe identical safe token is omitted in to the safe previously (0.064 nats), but retained in The safe will (3.614 nats).The selection depends on the prefix rather than assigning a fixed importance to an object-name token.
3: Protocol markersEight of twelve tokens overlapping the protocol tags are omitted from the entropy average.Across different contexts, knowing how to format a response does not imply equal certainty about which action to take.
4: An explicit alternative in the reasoning textThe phrase either take one of them and put it in microwave, or ... introduces alternative plans; the token either is retained (2.319 nats).The retained positions include a visible alternative-plan construction, rather than only final action names or arguments.
5: The same punctuation token in different contextsThe comma after At step 27 is omitted (0.041 nats), while the comma after Since this completes your action is retained (2.539 nats); both have the same token ID.Punctuation is not categorically treated as irrelevant: its predictability varies with the preceding context.

Auxiliary comparison with a retention-matched random mask (Appendix F.2). On 13,096 token-aligned responses from 134 ALFWorld games, we compare Top80 with a uniform random mask retaining the same number of tokens per response. The table focuses on protocol-tag and pure-whitespace tokens, identified from generated text independently of entropy; rates are pooled by category. Random80 gives the analytical expected omission rate.

Formatting categoryOmitted from the entropy average by Top80 (%)Expected omission under Random80 (%)
Protocol-tag tokens, e.g. <think>, </think>, <action>, </action>46.019.5
Pure whitespace, including spaces and line breaks40.219.6

Top80 omits these formatting realizations more often than a matched random mask. This is an auxiliary observation about estimator inputs, not a semantic-relevance test or a category-based selection rule. The target remains context-dependent step decision pressure.

The five cases are illustrative, not a representative estimate of semantic-selection accuracy. They support a context-sensitive interpretation and show why token categories alone are insufficient. We will describe the mechanism as uncertainty estimation motivated by reducing dilution from predictable continuations, without claiming every omitted token is semantically irrelevant. Whether Top-80% better matches independently identified decision positions remains open; these cases do not establish an oracle comparison.

Finally, exclusion concerns only entropy averaging. All valid response tokens, including underlined tokens, remain in the policy loss and share their step's modulated advantage. Exclusion therefore does not mean they are unnecessary or receive no update. We also distinguish this case evidence from the arithmetic fact that Top80 raises the selected-token average; that increase alone is not evidence of semantic relevance.

Q3 (Question 1): Can the authors revise the theoretical conclusion to distinguish an error-bound improvement from an actual error ordering, and state what is guaranteed for a nontrivial shared K?

A3: Yes. As detailed in A1, we will distinguish the Lemma's oracle uncertainty-estimation error comparison from the Theorem's bound on normalized modulation error. The latter controls each weight's multiplicative deviation from the same-prior oracle, but does not order the realized modulation errors of two estimators.

For the nontrivial shared-K case, A1 gives an additional sufficient condition: a uniform oracle gap Ut−H^tall≥Δ>0U_t-\widehat H_t^{\mathrm{all}}\ge\Delta>0 and token-entropy range at most RHR_H imply that any shared K≥1−Δ/RHK\ge1-\Delta/R_H satisfies H^tall≤H^tK≤Ut\widehat H_t^{\mathrm{all}}\le\widehat H_t^K\le U_t. Requiring K≤1−1/mmin⁡K\le1-1/m_{\min} ensures actual truncation at every step, provided the resulting interval is nonempty. Under these conditions, the estimator error is strictly reduced and the corresponding modulation-error bound tightens under the Theorem's regularity assumptions. This proposed sufficient-condition statement does not certify the empirical 80–90% range. The case study in A2 illustrates filtering behavior and does not supply the oracle labels or gap assumptions needed for this guarantee.

Q4 (Question 2): What evidence connects the selected tokens to semantic relevance, beyond entropy and downstream scores? A controlled comparison or narrower interpretation would materially affect my assessment.

A4: The revised manuscript now includes the five-case study in Appendix F.1 and the Top80-versus-Random80 formatting diagnostic under matched retention budgets in Appendix F.2, as summarized in A2. Top80 excludes protocol-tag and whitespace tokens from the entropy average more often than Random80. We identify these token types directly from the generated text, independently of their entropy values.

The cases address the context dependence more directly: the same safe token, and the same comma token, are retained or excluded under different prefixes. Our primary objective is to estimate the step's semantic decision pressure conditioned on its current prefix, rather than to maximize retention of a particular lexical or syntactic category. The random-mask comparison controls the retention budget for the formatting diagnostic; it does not independently label decision-relevant positions. We therefore distinguish this auxiliary evidence and the concrete cases from a verified fixed-K advantage relative to the oracle uncertainty. All valid response tokens remain in the policy loss, including those omitted from the entropy average.

Reviewer_ecFNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on credit assignment for RL-trained LLM agents in long-horizon interactive tasks. To improve upon the uniform credit allocation in group-based policy optimization methods, the authors introduce a Multi-Granularity Modulated (MGM) framework. MGM combines token-level Top-K_ _entropy filtering (for step uncertainty estimation), step-level temporal modulation, and task-level difficulty weighting to adjust policy gradients. The authors integrate MGM into GRPO and DAPO, reporting experimental improvements across ALFWorld, WebShop, and Search-Augmented QA benchmarks.

Critical strengths
  • Simple, plug-and-play method with comprehensive implementation details.** MGM requires neither process reward models (PRMs), additional annotations, nor inference-time search. It solely modifies the weighting of advantage values and can be directly integrated into GRPO or DAPO. Appendix D provides the pseudocode for four algorithms, while Tables 4 and 5 list the key hyperparameters, making the method easy to reproduce and adapt from an engineering perspective. For practitioners training RL agents, such low-cost modifications offer practical value.
  • **Consistent reported improvements across diverse settings.**In the reported results, MGM consistently outperforms the corresponding baselines across ALFWorld and WebShop, GRPO and DAPO, as well as Qwen2.5-1.5B and 7B; it also achieves a positive gain of 3.1 points on Llama-3.2-3B (Table 11). The reported gains over the direct baseline EMPG are also substantial—for instance, improving DAPO-1.5B on ALFWorld from 88.1 to 95.3, and GRPO-1.5B from 73.7 to 90.6 (Table 1). If these results hold across multiple random seeds, the empirical conclusion that "compositional advantage weighting is effective" would be meaningful.
Critical weaknesses
  • The model architectures evaluated throughout the paper are quite outdated. The authors have not applied the proposed method to the latest architectures, such as Qwen3 or Qwen3.5, to demonstrate its generalizability.
  • While I agree with the statement that 'Entropy-based modulation is therefore a principled way to stabilize sparse-reward training, but only if the entropy used for modulation faithfully estimates decision uncertainty rather than low-level surface-form uncertainty,' I fail to see the connection between this insight, the proposed method, and the theoretical results in Lemma 3.1 and Theorem 3.2. None of the design choices in the proposed method are actually derived from these two proofs; components such as top-k_ selection, S(t), and d(x) all appear to be purely empirical/heuristic choices. If the authors still claim that their theoretical analysis offers a meaningful contribution, please attempt to theoretically derive the optimal value of k_k for the top-k__ selection based on the proposed theory.
  • Uncited Baseline Sources and Potentially Unfair Comparisons: The baseline results in Table 1 appear to be directly copied from the EMPG and GiGPO papers without proper attribution, rather than re-evaluated as implied by the text. While citing published numbers is acceptable, the authors fail to ensure consistent experimental settings and random seed protocols (the baselines report 3-seed averages, whereas MGM's seed count is unspecified). More critically, the two source papers report conflicting results for the 1.5B GRPO baseline (72.8 in GiGPO vs. 65.6 in EMPG). By selectively adopting the lower value (65.6), the paper inflates its claimed maximum improvement in the abstract to 25.0 points, rather than 17.8 points.
  • Calculation Error: In Table 2, the actual average of the seven metrics for "Ours" is 45.03, yet it is misreported as 45.9 (whereas all baseline averages are correct). Table 2 omits GRPO and EMPG baselines, comparing only GRPO+MGM against GiGPO, which makes it impossible to isolate MGM's actual gains. Moreover, Search-QA has at most 4 interaction turns, poorly aligning with the "long-horizon" motivation. n Table 7 (WebShop, K=80%), the result (83.3/63.3) is identical to the full GRPO+MGM in Table 1. If step-level and difficulty modulations were disabled as claimed, this implies they have zero contribution on WebShop; otherwise, the actual experimental setup contradicts the stated protocol.
  • The conclusions drawn from the ablation studies are overstated. Table 10 claims to provide a "complete factorial breakdown," yet it contains only five rows and misses combinations such as S+D. Furthermore, the individual gains of the three components are 8.3, 10.6, and 7.5 (summing to 26.4), whereas the full method yields a gain of only 14.5. This clearly sub-additive effect—even accounting for ceiling effects near 100%—contradicts the claims of "orthogonality" and operating "without overlapping interference." Finally, the main text in Section 4.4 describes "adding either Step Index or Task Difficulty," which contradicts the cumulative addition indicated in the caption of Table 3.
Questions for authors
  • Are the 162 baseline numbers in Table 1—which are digit-for-digit identical to those in the EMPG paper—re-evaluated or directly copied? If copied, why were the notes regarding random seeds and sources in EMPG's original caption omitted?
  • Table 4 states that the 1.5B model uses 4×RTX 6000 Ada GPUs, whereas Appendix E.1 claims A100 GPUs were used. How many seeds were used for MGM, and were its results obtained under the exact same hardware and seed protocols as the EMPG baselines? Please also provide the standard deviations for all "with Ours" rows.
  • Please report a complete combinatorial ablation study (EMPG → EMPG+Top-KK→ EMPG+S → EMPG+D) under identical hardware and seed settings, and include the EMPG baseline on the Search-QA benchmark.
  • Section 4.5 claims that EMPG proved directly applying the 80/20 rule to multi-turn tasks "can be detrimental to training." However, EMPG's actual conclusion in Section 5.3 and Figure 3 is that steps with low initial entropy can still experience significant entropy changes during training ("a confident step can still require significant policy updates"), meaning token-level observations do not directly transfer to the step level. Please provide the exact textual basis for your claim. Moreover, EMPG's finding implies that low-entropy content is not negligible noise, which directly contradicts MGM's premise of discarding low-entropy tokens as "syntactic noise." Please discuss this discrepancy.
6 comments

We thank the reviewer for recognizing MGM’s implementation simplicity and the breadth of the reported experiments.

Q1 (Weakness 1): The model architectures evaluated throughout the paper are quite outdated. The authors have not applied the proposed method to the latest architectures, such as Qwen3 or Qwen3.5, to demonstrate its generalizability.

A1: Thank you for this suggestion. Our model selection was guided by the goal of evaluating MGM's incremental contribution as a post-training method. We selected Qwen2.5 at multiple sizes and Llama-3.2-3B as established backbones from different model families with conventional full-attention Transformer architectures. Comparing MGM with the corresponding optimization baselines on each backbone allows us to assess its contribution, while the two model families help examine whether the benefits extend beyond a single family.

We agree that evaluation on newer backbones is valuable. Qwen3.5 introduces a hybrid linear/full-attention architecture and a different upstream training recipe, including large-scale reinforcement learning [2]. We therefore test whether MGM also improves performance on Qwen3.5-2B. We adapted the existing verl-agent [1] codebase to support Qwen3.5-2B and set up a controlled comparison on ALFWorld with Qwen3.5-2B, evaluating GRPO, GRPO + EMPG, and GRPO + MGM (Ours) under the same protocol. All three runs use one random seed on two NVIDIA A100 GPUs.

Table R1. Completed ALFWorld evaluation with Qwen3.5-2B. All metrics are success rates (%), covering the six task categories and the overall evaluation (All). Results are reported for one seed per method.

MethodPickLookCleanHeatCoolPick2All
GRPO83.983.368.455.076.985.075.8
GRPO + EMPG85.781.8100.054.575.057.980.5
GRPO + MGM (Ours)84.684.693.1100.081.058.888.3

This completed comparison tests whether MGM's benefits extend to Qwen3.5-2B on ALFWorld. The overall success rates are 75.8% for GRPO, 80.5% for EMPG, and 88.3% for MGM. MGM improves over GRPO by 12.5 percentage points and over EMPG by 7.8 points, with gains on five of the six task categories relative to GRPO; Pick2 is lower for MGM in this run. We will scope the generalization claim to the evaluated backbone and task, and further extend agentic RL evaluations to larger and newer backbones in future work.

Reference

[1] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-Group Policy Optimization for LLM Agent Training. arXiv:2505.10978, 2025.

[2] Qwen Team. Qwen3.5-2B: Official Model Card.

Q2 (Weakness 2): While I agree with the statement that 'Entropy-based modulation is therefore a principled way to stabilize sparse-reward training, but only if the entropy used for modulation faithfully estimates decision uncertainty rather than low-level surface-form uncertainty,' I fail to see the connection between this insight, the proposed method, and the theoretical results in Lemma 3.1 and Theorem 3.2. None of the design choices in the proposed method are actually derived from these two proofs; components such as top-k_ selection, S(t), and d(x) all appear to be purely empirical/heuristic choices. If the authors still claim that their theoretical analysis offers a meaningful contribution, please attempt to theoretically derive the optimal value of k_k for the top-k__ selection based on the proposed theory.

A2: Thank you for asking us to clarify the connection between the analysis and the method. The analysis quantifies how accurately uncertainty must be estimated to control distortion of MGM's normalized credit weights. This supplies a theoretical rationale for improving the estimator used by the modulation, while the particular temporal and difficulty priors are motivated design choices evaluated empirically.

The connection to MGM. Lemma 3.1 compares the Top-K estimator with all-token averaging relative to the defined semantic oracle UtU_t: for a retention choice covered by the existence statement, ∣H^tK−Ut∣≤∣H^tall−Ut∣|\widehat H_t^K-U_t|\le|\widehat H_t^{\mathrm{all}}-U_t|. Theorem 3.2 connects this estimation error to the normalized multiplier actually analyzed in the paper. Under its Lipschitz assumptions, with the same positive priors Bt=S(t)D(x)B_t=S(t)D(x) held fixed,

\le 2L\left\lVert\widehat H-U\right\rVert_\infty.$$ The bound has a concrete interpretation. If $\varepsilon=\|\widehat H-U\|_\infty$, then every normalized credit weight satisfies $$e^{-2L\varepsilon}\le \frac{M_t(\widehat H)}{M_t(U)} \le e^{2L\varepsilon}.$$ Improving the estimator contracts this guaranteed range of multiplicative distortion relative to the same-prior oracle. Moreover, although mean normalization couples the weights across the batch, the bound has no additional factor proportional to the number of steps or the ratio of the largest to smallest positive prior. The analysis thus connects the token-level estimator to the composed modulation with $S(t)$ and $D(x)$; it does not uniquely derive their exponential forms or establish their empirical benefits. This approximation guarantee motivates improving the estimator without requiring a universal numerical optimum for K. We will make this interpretation explicit. The sentence following Theorem 3.2 currently states that Top-K produces an allocation "closer to the oracle semantic allocation than all-token entropy," and the appendix proof ends with the same comparison. We will revise these two sentences to describe the tighter distortion bound: its contraction is meaningful even though it does not order the two realized modulation errors. Downstream benefits are evaluated experimentally. **How K enters the analysis.** To further connect the theory to the choice of retention ratio, we will make its dependence on K explicit through an additional sufficient-condition result. Suppose every step satisfies $U_t-\widehat H_t^{\mathrm{all}}\ge\Delta>0$, and its token-entropy range is at most $R_H$. Then $$0\le\widehat H_t^K-\widehat H_t^{\mathrm{all}} \le(1-K)R_H,$$ so any shared $K\ge1-\Delta/R_H$ gives $\widehat H_t^{\mathrm{all}}\le\widehat H_t^K\le U_t$. The oracle estimation error is strictly reduced wherever $\lceil Km_t\rceil<m_t$. If every step must undergo truncation, a sufficient shared interval is $1-\Delta/R_H\le K\le1-1/m_{\min}$, where $m_{\min}=\min_t m_t$, provided the interval is nonempty. This proposed additional result gives a conservative condition for shared truncation, not an optimality claim or a claim that our experimental batches satisfy its assumptions. Because $\Delta$ depends on the latent oracle, it is not a directly observable tuning rule. Deriving a K that maximizes downstream training performance would additionally require a model of the optimization dynamics and task rewards, which the present approximation analysis does not provide. The reported 80–90% range is therefore empirical evidence about sensitivity. The theoretical contribution is the explicit estimator-to-modulation connection above, whose validity does not require a universal numerical optimum for K.

Q3 (Weakness 3): Uncited Baseline Sources and Potentially Unfair Comparisons: The baseline results in Table 1 appear to be directly copied from the EMPG and GiGPO papers without proper attribution, rather than re-evaluated as implied by the text. While citing published numbers is acceptable, the authors fail to ensure consistent experimental settings and random seed protocols (the baselines report 3-seed averages, whereas MGM's seed count is unspecified). More critically, the two source papers report conflicting results for the 1.5B GRPO baseline (72.8 in GiGPO vs. 65.6 in EMPG). By selectively adopting the lower value (65.6), the paper inflates its claimed maximum improvement in the abstract to 25.0 points, rather than 17.8 points.

A3: Baseline provenance and the GRPO reference. Thank you for carefully checking the baseline sources and identifying the different GRPO results reported across the two papers. All non-MGM entries in our Table 1 were taken consistently from EMPG's reported baseline set, including its GRPO result of 65.6. We did not assemble the table by choosing between EMPG and GiGPO results for individual baselines. The 65.6 entry follows the same source rule as the other baseline entries; it was not selected over 72.8 because it produces a larger improvement. Thus, the difference between the two published GRPO results does not reflect the selective adoption suggested in the review.

The source-attribution sentence was removed during final editing to meet the page limit. We have restored it in the revised Table 1 caption, which now explicitly states that the remaining results are adopted from EMPG and distinguishes those entries from our MGM results.

Additional seeds and uncertainty reporting. The submitted MGM entries were single-run results. We have supplemented the Qwen2.5-1.5B experiments and updated both "with Ours" rows to report means and sample standard deviations across three random seeds (n=3n=3, with denominator n−1n-1). The table below compares the submitted entries with the values in the current revised manuscript. All metrics use a 0–100 scale; ALFWorld metrics and WebShop success rates are percentages.

Table R2. Submitted and revised Qwen2.5-1.5B MGM results. Revised entries are mean ± sample standard deviation.

MethodVersionPickLookCleanHeatCoolPick2AllWebShop ScoreWebShop Success
GRPO + MGMSubmitted97.291.784.281.895.585.790.683.363.3
GRPO + MGMRevised94.6 ± 2.371.0 ± 18.292.0 ± 7.987.7 ± 6.095.5 ± 4.579.3 ± 6.088.3 ± 2.183.4 ± 0.263.2 ± 1.4
DAPO + MGMSubmitted100.0100.0100.090.996.280.095.390.478.9
DAPO + MGMRevised100.0 ± 0.095.4 ± 4.0100.0 ± 0.095.1 ± 4.690.9 ± 4.890.0 ± 10.096.1 ± 1.489.1 ± 1.577.1 ± 1.6

The reported sample standard deviations of aggregate success rates are 2.1 and 1.4 percentage points on ALFWorld, and 1.4 and 1.6 points on WebShop, for GRPO+MGM and DAPO+MGM respectively. Some task categories exhibit larger variation, which is visible in the table. Both revised rows retain positive aggregate margins over the corresponding GRPO/DAPO and EMPG reference values. We report the complete updated rows, including metrics that decrease relative to the submission, rather than retaining the better of the original and updated value for each metric.

Given our current computational budget, the Qwen2.5-7B entries remain single-run results and are labeled accordingly. The relatively small sample standard deviations of the aggregate 1.5B metrics provide preliminary evidence for the consistency of our evaluation results at that scale. We are committed to supplementing the Qwen2.5-7B results with three-seed experiments and reporting the corresponding means and sample standard deviations once sufficient computational resources become available. We have also updated the abstract's ALFWorld gains to match the revised means: up to 22.7 percentage points over GRPO and 15.3 points over DAPO.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.