acceptodds
Withdrawn from ICLR 2027

Know Your Tokens: Structure-Aware Gradient Allocation for Agentic Search via Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement Learning (RL) from verifiable rewards is a powerful paradigm for training Large Language Model (LLM)-based search agents. Applying RL to agentic search models faces a core technical bottleneck: (a) standard Group Relative Policy Optimization (GRPO) causes gradient instability under parameter-efficient training (LoRA), and (b) existing fixes that subsample tokens by position (Stochastic GRPO, S-GRPO) restore convergence but discard gradient signal from late-position decisions that are equally critical. We show that position is not a well suited proxy for importance in multi-step search trajectories where critical decisions can occur at any position. We propose SA-GRPO (Structure-Aware S-GRPO), a general methodology for training LLM-based search agents via RL that addresses both problems: each model-generated token receives an inclusion probability in the policy gradient loss based on its functional role in the trajectory; search decisions and queries always receive gradient; answer tokens at high probability; reasoning is subsampled; and redundant re-searches are rarely included. Role assignment requires only deterministic parsing of tool-use tags, adding zero compute overhead. We evaluate on the HotpotQA and MuSiQue multi-hop Question Answering (QA) benchmarks using Mistral-7B-Instruct and Qwen2.5-7B-Instruct with QLoRA (NF4 quantization with LoRA adapters), comparing the QLoRA baseline against four RL-trained arms: GRPO, S-GRPO, SA-GRPO, entropy-weighted baseline under identical training conditions. Using a normalized answer-quality score (0-1) on Mistral/QLoRA, all subsampled arms improve over the baseline (0.620) while full GRPO barely moves (0.642). On HotpotQA, S-GRPO leads (0.703) with SA-GRPO close behind (0.684, 50% fully correct vs 40% baseline), under the same expected token budget (). On genuine 2-search trajectories under Qwen, where role-based selection provides maximum benefit, SA-GRPO scores 0.820 versus S-GRPO's 0.777. On the harder compositional questions of MuSiQue, SA-GRPO leads in both regimes: under the resource-constrained Mistral setting it is the only arm to improve over the baseline (0.416 vs 0.349, McNemar vs base), while GRPO falls below base and the entropy baseline collapses; under the stronger Qwen base all arms improve and SA-GRPO scores highest (0.590 vs 0.517) though within noise of GRPO (0.579). Beyond the method, we identify a key empirical finding: role-based selection requires multi-hop trajectory diversity to differentiate from position-based selection, a design guideline with implications for evaluation design in agentic RL.

Primary area:
generative models
Keywords:
GRPO, RL, Multihop, SGRPO

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.17 over 6 reviews, above 2% of the others.
Reviewer_ZmQKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SA-GRPO, which samples tokens for the policy-gradient objective according to their functional roles in an agent trajectory. Search decisions and novel queries receive high inclusion probabilities, while answers, reasoning, and repeated queries receive different probabilities under an expected token budget. Experiments compare full GRPO, position-based sampling, role-based sampling, and an entropy baseline on two models and two multi-hop QA datasets. Analyses include overall performance and subsets defined by the number of generated searches.

Critical strengths
  1. The method is simple and exploits readily available trajectory structure. Tool-use tags provide role information without requiring a separately trained importance estimator.
  2. The abstract and main tables report unfavorable comparisons, such as S-GRPO leading on HotpotQA. However, some later claims are stronger than the evidence, including “confirms the mechanism” in Appendix C.2.
Critical weaknesses
  1. The experiments do not distinguish role weighting from stochastic sparsification. Role-dependent inclusion changes the contribution of different tokens to optimization. Without appropriate importance correction, it should not be interpreted as an unbiased estimate of the original uniformly weighted objective. A deterministic role-weighted baseline is missing, so the benefit cannot be attributed specifically to selecting fewer tokens. The motivating claim that GRPO is unstable under LoRA also lacks direct diagnostics and sits uneasily with the paper’s own results: under the same QLoRA configuration, GRPO on Qwen scores 0.763 on HotpotQA and 0.579 on MuSiQue, above S-GRPO on both. In addition, SA-GRPO’s non-mandatory inclusion probabilities are higher than S-GRPO’s, and the budget scaling applies only when the expected count exceeds k. Realized token counts may therefore differ across arms, but they are not reported.
  2. The main multi-hop subgroup evidence is affected by selection, while overall superiority is inconsistent. "Exactly two searches" is determined by each model's behavior, so the compared subsets contain different questions and sample counts. Their score difference is not a matched comparison on a common multi-hop test set. Overall, SA-GRPO does not lead on HotpotQA and has the lowest score among the RL arms on Qwen (0.759). The strongest result, Mistral/MuSiQue, is evaluated at step 200 rather than after a full pass, without an explained selection rule. Its significance test is against the base model, with no direct test against S-GRPO reported, so “the only arm to improve” does not establish that SA-GRPO beats S-GRPO. The paper does not mention multiple training seeds, and the bootstrap confidence intervals described in the methods do not appear in the results.
Questions for authors
  1. How was step 200 chosen for Mistral/MuSiQue, and did it apply to all arms?
  2. Are the improvements over S-GRPO stable across training seeds, and what is their direct comparison with confidence intervals?
  3. Does stochastic role sampling outperform deterministic role-weighted GRPO on a common test subset defined in advance by task properties?
Reviewer_poaPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes SA-GRPO, a structure-aware token selection strategy for reinforcement learning of search agents. It extends stochastic GRPO by selecting tokens according to their functional role rather than position: search decisions and novel queries are always included in the policy loss, while answers, reasoning, and redundant searches receive progressively lower inclusion probabilities. Roles are assigned through deterministic parsing of tool-use tags, without additional model calls.

Its main contributions are a lightweight modification to token selection within GRPO and an empirical investigation of when that modification helps. The central hypothesis is that preserving training signal for later search actions matters more in difficult, multi-step retrieval tasks.

Validation compares an untrained baseline and four RL variants—GRPO, S-GRPO, SA-GRPO, and entropy-based selection—using Mistral-7B and Qwen2.5-7B with QLoRA on HotpotQA and MuSiQue, with 300 evaluation questions per setting. SA-GRPO’s strongest result is on Mistral/MuSiQue, improving judge score from 0.349 to 0.416, versus 0.358 for S-GRPO. It also achieves the highest MuSiQue score with Qwen, although its advantage over GRPO is not statistically established. It does not lead on HotpotQA overall.

The results support conditional benefits, particularly on harder multi-hop tasks, rather than general superiority. Two-search subset results are suggestive but compare potentially different question subsets. Missing gradient diagnostics, probability ablations, and repeated-seed results limit claims about the mechanism and robustness.

Critical strengths
  • A simple, well-motivated use of agent structure. SA-GRPO uses explicit tool-use roles to preserve training signal for search decisions and queries regardless of their position. This directly addresses a limitation of position-based sampling while requiring only deterministic parsing and no additional model calls. Its practical simplicity and clear connection to multi-step agent behavior constitute a meaningful methodological strength.
  • Evidence of benefit in a challenging setting. On Mistral/MuSiQue, SA-GRPO improves answer-quality score from 0.349 to 0.416, compared with 0.358 for S-GRPO and 0.331 for GRPO, alongside an increase in fully correct answers from 13% to 20%. The reported significant improvement over the base provides concrete evidence that role-aware selection can make RL more effective where the alternatives offer little benefit or degrade performance. This is the strongest empirical consideration supporting acceptance, although it establishes a setting-specific advantage rather than general superiority.
Critical weaknesses
  • The experiments do not establish the claimed mechanism. The paper motivates SA-GRPO as addressing LoRA gradient instability, but provides no gradient diagnostics or training-stability curves. Its two-search analysis also compares subsets selected by each policy’s behavior, potentially containing different questions. Consequently, the results cannot isolate whether gains arise from protecting later search tokens, altered search behavior, or differences in subset difficulty. This substantially weakens the central mechanistic contribution.
  • Evidence of superiority over competing methods is limited. SA-GRPO does not lead on HotpotQA, and its Qwen/MuSiQue advantage over GRPO is not statistically established. The strongest result is concentrated in Mistral/MuSiQue at a deliberately shortened training horizon, without reported repeated-seed results or sensitivity analysis for the hand-set selection probabilities. Although improvement over the base is significant, that does not establish superiority over competing RL methods. These limitations leave the robustness and broader value of the proposed modification insufficiently demonstrated.
Questions for authors

Please refer to the Weaknesses

Reviewer_hwh5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper changes how S-GRPO picks which tokens go into the loss: instead of using position, it parses the search/answer tags and gives each token type a fixed hand-set probability. Experiments on HotpotQA and MuSiQue with two 7B models under QLoRA show mixed results against S-GRPO and GRPO.

Critical strengths
  1. The motivation is reasonable (position is a poor proxy for importance in multi-step search trajectories)
  2. The method is easy to implement
Critical weaknesses
  1. Limited novelty, and the core design is untested. The only change from S-GRPO is replacing the position rule with a table of hand-set probabilities per token type. The authors state these are heuristics without sensitivity analysis, so there is no evidence that these values, or role-based selection itself, are what matter. Since this table is the entire contribution, the lack of ablation is critical.

  2. The experiments cannot attribute the gains to role-based selection. With short outputs, the k = 100 budget rarely binds, and SA-GRPO gives most tokens P ≥ 0.5 versus 0.3 in S-GRPO, so it likely puts more tokens into the loss. The paper does not report token counts or compare against a random baseline with a matched token count. The results are also mixed: S-GRPO wins on HotpotQA, SA-GRPO ties with GRPO under Qwen, and only one setting (Mistral/MuSiQue) shows a clear win. No seeds or variance are reported.

Questions for authors
  1. What is the average number of tokens per rollout that enter the loss for each arm? If SA-GRPO uses noticeably more tokens than S-GRPO, the comparison is confounded.
  2. Where does P = 0.3 come from? I could not find this value in Lee & Tong (2025), and that paper also does not state that α = 50 was used for SVAMP. Several references also have incorrect author names (e.g., arXiv:2504.20834 is by Alan Lee and Harry Tong).
  3. For Mistral/MuSiQue, were all arms evaluated at step 200, or only SA-GRPO? How was this stopping point chosen?
  4. How many seeds were run? Are the differences between arms (e.g., 0.590 vs 0.579) stable across seeds?
  5. Because the role probabilities differ and there is no inverse-probability (Horvitz–Thompson) correction, SA-GRPO effectively reweights the loss by role. How is this consistent with the claim that the objective is identical to S-GRPO?
Reviewer_k4ueOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies RL training of LLM search agents under QLoRA. It builds on S-GRPO, which stabilizes GRPO under LoRA by subsampling the tokens that enter the policy-gradient loss according to their position, and argues that position is a poor proxy for importance in multi-step search, where a second-hop query can appear late in the trajectory. It proposes SA-GRPO, which parses tool-use tags to assign each generated token a functional role and includes it with a hand-set probability: 1.0 for search decisions and queries, 0.8 for answers, 0.5 for reasoning and 0.1 for redundant re-searches, with the objective and token cap unchanged.

Experiments train Mistral-7B and Qwen2.5-7B with QLoRA on 1,350 questions from HotpotQA and MuSiQue, retrieving from each question's own distractor paragraphs, and score answers with a three-LLM judge council. SA-GRPO does not lead on HotpotQA, where S-GRPO is best under Mistral and all arms are within noise under Qwen. On MuSiQue it scores highest under both models, significantly above the base under Mistral (0.416 vs. 0.349) and statistically tied with GRPO under Qwen. The paper concludes that role-based selection helps where trajectories contain genuine multi-hop search.

Critical strengths
  1. A simple and practical observation. Position is indeed a poor proxy for token importance in tool-using trajectories, and assigning roles needs only a tag parser, so the method is cheap to adopt in existing pipelines. This is the main reason the paper is of interest.

  2. A controlled comparison with candid limitations. The arms share one configuration and differ only in the selection rule, results cover two models and two datasets with paired McNemar tests and Holm correction, and the limitations section openly acknowledges the hand-set probabilities and the scarcity of multi-hop trajectories.

Critical weaknesses
  1. The premise and mechanism are not established. The paper takes from Lee & Tong (2025) that full-token GRPO is unstable under LoRA, but that workshop paper itself attributes GRPO's failure to possible overfitting or insufficient tuning, and no gradient norms or training curves are shown here. The Qwen results, where GRPO matches SA-GRPO on MuSiQue (0.579 vs. 0.590), do not support this as a general premise. Nor is it shown why reweighting tokens by hand-set role probabilities should help: there is no role ablation or diagnostic, and the 2-search analysis compares subsets selected by each trained policy (25 to 200 trajectories per arm), so it cannot isolate the mechanism.

  2. The evidence does not establish a reliable gain over S-GRPO. The central comparison is never tested directly: the McNemar tests compare each arm with the base on binary %Good, and the promised bootstrap intervals are not reported. No repeated runs or variance are reported, and the results are inconsistent across settings, with SA-GRPO behind S-GRPO on HotpotQA and level with GRPO on MuSiQue/Qwen. The token budget is only a cap, and SA-GRPO's higher inclusion probabilities (0.8 and 0.5 for answers and reasoning, against 0.3) likely put more tokens into the loss, so how many tokens is confounded with which tokens. The narrow setting, retrieval over each question's own 10–20 paragraphs on two benchmarks without established search-agent RL baselines [1–3], further limits what the results can show. Table 1 marks query tokens at positions 5–15 with P = 1.0 despite α = 10; R_format = 0.5 is undefined; related work appears only as notes in Appendix D; and the generalization to "any tool-using LLM agent" is unsupported.

References

  1. Jin et al. Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning. https://arxiv.org/abs/2503.09516
  2. Song et al. R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning. https://arxiv.org/abs/2503.05592
  3. Chen et al. ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning. https://arxiv.org/abs/2503.19470
Questions for authors
  1. Is SA-GRPO reliably better than S-GRPO? Please report direct paired tests between the two, the promised bootstrap intervals, and the mean and standard deviation over at least three seeds. Does the MuSiQue/Mistral gain survive?

  2. How many tokens enter the loss per rollout in each arm? S-GRPO caps the realized count (Eq. 3) while SA-GRPO caps its expectation (§3.3); a control that matches realized counts would separate how many tokens from which tokens.

  3. What evidence shows that GRPO is unstable under LoRA in this setting? Gradient-norm or reward curves, and results at a higher LoRA rank, would test the premise.

  4. Generality. Results with open-corpus retrieval and a Search-R1-style baseline, or on further multi-hop datasets such as 2WikiMultiHopQA [1] and Bamboogle [2], would show whether the gain extends beyond this setting.

  5. Implementation and reporting. How does the entropy baseline map entropy to inclusion probabilities and control its budget? Is the loss normalized over all tokens or the selected subset? Figure 1 lists a different LoRA rank from §4.3; which is correct? Please also add F1 and EM to the main tables.

References

  1. Ho et al. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps (2WikiMultiHopQA). https://aclanthology.org/2020.coling-main.580/
  2. Press et al. Measuring and Narrowing the Compositionality Gap in Language Models (Bamboogle). https://arxiv.org/abs/2210.03350
Reviewer_ucG4OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SA-GRPO. A Structure-Aware Gradient Allocation RL method for agentic search tasks.

Critical strengths

Variables are controlled in the experiments, and paper are honestly written.

Critical weaknesses
  1. In line 277-279, says "Effect sizes are reported as paired bootstrap 95% confidence intervals on llm norm differences (10,000 resamples)." However, 95% CI are not reported. Did the author purposely removed the CI? In the main experiment table, the Qwen models on HotPotQA's results are within a 3% gap, without reported CI, it is hard to know whether the results are significant or just pure noise.
  2. The related work is put in Appendix D. somehow. while there are still 1+ pages till the 9 page limit. And, the related work does not mention other RL methods in agentic search domain, like ARPO.
  3. in line 306-307, the Mistral training early stopped at step 200, and show a large gain compare to other baselines. Where you say it is "by design" without further explanation. I wonder what design that is? It is hard to convince me that this is not a "cherry-pick" of training steps.
Questions for authors

See weakness please.

Reviewer_ktdWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces SA-GRPO (Structure-Aware Stochastic GRPO), a role-based token selection strategy designed to stabilize and improve the Reinforcement Learning (RL) training of Large Language Model (LLM) search agents under parameter-efficient regimes like QLoRA.

SA-GRPO replaces position-based token selection heuristics with functional role-based token inclusion probabilities derived deterministically from tool-use tags (and ) and Jaccard query overlap. It identifies a critical regime boundary, demonstrating that trajectory action diversity (multi-hop behavior) is a necessary condition for role-based token selection to differentiate itself from position-based selection.

Experiments show that SA-GRPO beats GRPO and S-GRPO on two multi-hop QA benchmarks under a resource-constrained Mistral/QLoRA setting.

Critical strengths
  • Effective addressing of gradient instability in parameter-efficient RL. The paper correctly identifies that standard GRPO causes gradient instability under LoRA, and that existing position-based fixes improperly penalize late-stage critical decisions. The proposed role-based allocation is intuitive and computationally zero-cost.
  • Novel utilization of agentic structure. Leveraging the inherent tool-use and trajectory structure of search agents to govern token-level gradient allocation provides a principled alternative to naive positional heuristics.
Critical weaknesses
  • Reliance on hand-set heuristics. The inclusion probabilities for different token roles (e.g., 1.01.0 for queries, 0.80.8 for answers, 0.50.5 for intermediate reasoning, 0.10.1 for redundant re-searches) are hand-crafted heuristics without a rigorous sensitivity analysis or ablation regarding their optimal values.
  • Insufficient experimental evaluation. The evaluation is limited to only two benchmarks (HotpotQA and MuSiQue) with a relatively small test sample size. This scope is insufficient to comprehensively substantiate the claims and demonstrate that the approach is truly "solid" across diverse retrieval and reasoning distributions.
  • Outdated backbone models and baselines. The choice of backbone models (Mistral-7B-Instruct-v0.3 and Qwen2.5-7B-Instruct) and competing baselines are somewhat outdated, making it difficult to gauge the method's competitiveness against the current state-of-the-art. Also, the paper lacks empirical comparisons against other prominent credit assignment or token-level optimization strategies in LLM reasoning, such as GiGPO, IGPO, and HiPER, leaving the relative efficacy of SA-GRPO unverified against contemporary alternatives.
Questions for authors
  • Generalization across broader benchmarks. To address concerns regarding evaluation scale and insufficiency, have the authors tested or planned to evaluate SA-GRPO on larger-scale, more diverse web-search or multi-hop QA benchmarks (e.g., all open-domain QA datasets used in Search-R1) to further prove its robustness?
  • Comparison with advanced credit assignment methods. How does SA-GRPO theoretically and empirically compare against alternative trajectory-level or token-level credit assignment frameworks (such as GiGPO, IGPO, or HiPER)? Could the authors provide benchmark comparisons against these methods?
  • Generalizability to other tool-use domains. Since the method relies on parsing structured tags like <search> and <answer>, how easily can this taxonomy be adapted to more free-form agentic environments (such as interactive coding or multi-API tool use) where explicit tag boundaries may be less cleanly defined or completely absent?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.