acceptodds
Under review as a conference paper at ICLR 2027

Learning at the Branch Point: State-Local Policy Optimization for Search Agents

Abstract

Reinforcement learning (RL) has emerged as an effective approach for training large language model (LLM)-based search agents on knowledge-intensive tasks requiring iterative information seeking. However, existing RL methods typically optimize at the trajectory level, where credit assignment based primarily on whole-trajectory outcomes can obscure the individual contributions of useful and redundant search actions. To address this limitation, we propose State-Local Policy Optimization (SLPO), which shifts the optimization unit from whole trajectories to individual actions under the same search history. At each candidate state, SLPO executes and scores alternative actions, deriving local advantages from their relative rewards for action-level updates. To explore diverse search directions while limiting rollout cost, we structure exploration as a semantic tree, clustering similar actions and expanding only cluster representatives while retaining all sampled alternatives for local comparison. We use precomputed evidence-to-document mappings to reward actions that retrieve new evidence without an LLM judge during training. We further prioritize states with greater within-state reward dispersion for more informative local comparisons. Across seven QA benchmarks, SLPO improves answer accuracy (COVER-EM) over the strongest baseline by 3.54 and 3.84 percentage points for 4B- and 8B-parameter models, respectively, while using comparatively few searches. These results demonstrate the effectiveness of state-local credit assignment for training LLM-based search agents.

Primary area:
reinforcement learning
Keywords:
LLM Search Agents, Agentic Reinforcement Learning, Credit Assignment, State-Local Policy Optimization, Semantic-Tree Exploration, Evidence-Grounded Process Rewards

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_bP4gOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses credit assignment over long trajectories in RL for search agents, and proposes SLPO. Through semantic-tree exploration, SLPO samples multiple candidate actions under the same search history. It then uses an offline-constructed evidence-to-document mapping to assign immediate transition rewards. Search actions are rewarded for newly acquired evidence. Answer actions are rewarded for a combination of correctness and evidence coverage. Experiments use two Qwen3 models of different scales and a validation mixture of seven QA datasets. The results show that SLPO improves COVER-EM over the strongest baseline and uses fewer search calls than some baselines.

Critical strengths
  1. The problem is practical and significant. RL with outcome rewards cannot distinguish the values of individual step-level actions, which is a long-standing problem in the field.
  2. Transparency and reproducibility are good. The paper provides extensive experimental details, which makes reviewing easier.
Critical weaknesses
  1. No independent final evaluation. The authors select and report the best checkpoint on the same evaluation mixture, so the reported scores may carry an optimistic bias from model selection. Applying the same procedure to all methods does not guarantee that this bias is equal across methods. This may distort the comparison with the baselines.
  2. Incompleteness. The paper still appears insufficiently complete, and the main text is too thin.
  3. Lenient primary metric. The evaluation mainly uses COVER-EM, which is far more lenient than EM and F1. It also carries the risk of inflating ("hacking") the score by outputting more content.
  4. Questionable results. Bamboogle has only 125 examples, so every score on it should be a multiple of 0.8%. GiGPO's scores of 23.39 (4B) and 73.79 (8B) are not, and it is unclear whether examples are missing or there is some other issue.
  5. Unequal privileged information. The method uses DeepSeek to construct additional privileged information, but most of the compared baselines do not have access to it. This makes the comparison insufficiently fair.
  6. Incomplete evidence mapping. The Wiki corpus may contain many documents supporting the same fact. A limited number of fixed retrievals can hardly exhaust all valid evidence documents. As a result, a valid search may be wrongly penalized because it misses the offline mapping.
  7. No semantic deduplication. The authors decide whether information is new based on atomic facts (evidence items) and their corresponding document IDs. However, they do not explicitly design a deduplication mechanism that accounts for semantic equivalence or entailment across evidence items. If two semantically equivalent facts are annotated as different evidence items, a repeated retrieval may be wrongly rewarded.
  8. Myopic rewards. The reward design considers only the gain from a single-step transition. For necessary exploratory queries, which cannot directly provide the evidence needed to answer the question, the framework cannot assign credit correctly.
  9. Heuristic design. The constants in the reward schedules for the two action types come with no explanation of how they were chosen. Neither the specific constants nor the relative scale between search rewards and answer rewards is adequately justified or supported by a sensitivity analysis.
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Follow-up Response to Reviewer bP4g

We appreciate the reviewer's continued consideration. Our earlier response addressed the nine concerns individually; here we clarify the central distinction behind our contribution and provide additional experimental evidence.

1. The Core SLPO Formulation vs. Its QA Reward Implementation

Most concerns address QA reward construction or evaluation, rather than state-local optimization itself. SLPO changes how multi-turn agents learn: each tool interaction provides a natural decision boundary. It independently executes alternatives under exactly the same search history, computes within-state advantages from observed rewards, and updates only the corresponding action tokens.

This representation also permits historical reasoning to be hidden during rollout and training, addressing context growth and training–inference mismatch identified in veRL's multi-turn documentation.

The evidence mapping provides verifiable QA rewards, not a mathematical requirement of state-local optimization. Concerns about its completeness should be distinguished from the soundness and originality of the training formulation.

2. The Evaluation Protocol Does Not Favor SLPO (W1)

We did not introduce the best-checkpoint rule to favor SLPO. It follows our strongest baseline, AT²PO, which explicitly reports the checkpoint with the highest average evaluation score (Sec. 5.1). The original Search-R1, GiGPO, PiCA, and AT²PO papers likewise do not report a separate held-out evaluation after checkpoint selection. We explicitly label our scores as validation results.

The 200-step window was chosen based on prior practice: GiGPO uses 200 iterations for QA, and AT²PO's published curve shows little further improvement around steps 200–220. We apply the same window, data, and metric to all reproduced methods. Figure 3 shows SLPO ahead even at fixed step 200 at both model scales, so its advantage does not depend on selecting a favorable peak.

3. Evidence Supervision Alone Does Not Explain the Gain (W5)

SLPO is not the only method using external process supervision. PiCA uses DeepSeek annotations to train a step-reward model, yet scores 61.00% in our unified 4B evaluation. Our Evidence Credit control uses the same mapping as SLPO but retains trajectory-derived advantages: 65.88% versus 68.34% for SLPO.

We also tested (i) reversing selected advantage signs using evidence feedback, which did not improve on Evidence Credit, and (ii) summing SLPO's step rewards into a trajectory reward for standard GRPO, which performed worse than outcome-only GRPO. Both results will be added to the revision. These controls show that richer supervision or trajectory-level reward changes alone do not reproduce SLPO's improvement.

4. Task-Driven Rewards and Intermediate Evidence (W6–W9)

Characterizing our reward as “myopic” overlooks its deliberate task-driven design. The guiding idea is to understand and decompose a task before designing local rewards. In search QA, successful answering requires supporting facts linking the question to its answer. We can therefore assess meaningful intermediate progress without waiting for the final outcome.

Given both question and correct answer, our annotator identifies atomic supporting claims rather than solving the question. An independent verifier checks that the claims jointly support the answer. These include necessary intermediate facts, not merely statements of the final answer. Retrieving them earns immediate credit even before the agent can answer. This is intentional: it recognizes useful actions without introducing uncertainty from later decisions.

Each claim also maps to multiple independently verified documents, but earns reward only upon first coverage (Appendix A.3, Eq. 6). Different documents supporting the same mapped fact cannot generate repeated rewards. This handles document-level redundancy, though exhaustive coverage and cross-claim deduplication are not guaranteed.

Some exploratory actions may have delayed benefits beyond mapped evidence. Adding downstream outcomes via Subtree Reward did not improve performance (65.38% vs. 68.34%, Table 3). Extending task-driven rewards to less directly verifiable settings remains worthwhile.

We believe these QA-specific concerns (addressed in our earlier response and further clarified above) do not materially undermine our main findings or establish a flaw in SLPO's state-local optimization. Our central contribution is to assign credit to individual decisions under the same history, rather than letting later trajectory outcomes obscure their contributions. The additional controls and experimental results support the value of this formulation beyond merely introducing richer reward annotations.

Thank you for the response. It clarified some of my concerns, but three issues remain. First, I understand that prior work has used the same evaluation protocol. However, this precedent does not resolve the potential selection bias introduced by selecting checkpoints and reporting performance on the same validation set. I would still like to see an evaluation on an independent held-out set.

Second, the authors offer a plausible explanation for the Bamboogle results. However, giving equal weight to a batch containing 124 Bamboogle examples and another containing just one is inconsistent with the per-example accuracy described in the paper. The response does not provide scores verified against the individual predictions.

Finally, the existing predictions could be rescored using exact match and token-level F1 to assess whether the main findings are robust to the choice of metric. This analysis requires no retraining, but the response does not report it. Given these unresolved concerns, I am maintaining my original rating.

AuthorsOpenReview ↗

Further Response to Reviewer bP4g

We thank the reviewer for the follow-up. We have reanalyzed our existing evaluation results to directly address the checkpoint-selection concern and would like to clarify the remaining issues.

1. Independent Evaluation

Both SLPO and AT²PO are trained on the HotpotQA training split. We now select checkpoints using only HotpotQA validation and report performance on the other six QA datasets, which are excluded from checkpoint selection.

The results are:

ModelSLPOAT²POImprovement
4B68.65%64.64%+4.01 pp
8B66.98%63.13%+3.85 pp

SLPO maintains substantial improvements at both model scales when checkpoint selection and final evaluation use separate datasets. The gains are consistent with our original findings, directly addressing the concern that SLPO benefits disproportionately from selecting checkpoints on the same evaluation mixture.

These results also demonstrate that SLPO's advantages extend to six QA benchmarks not used for RL training or checkpoint selection under this protocol.

2. GiGPO Aggregation

We agree that unweighted batch averaging differs from the stated per-example accuracy. This issue originates from the publicly released GiGPO implementation used in our reproduction, rather than a deliberate modification of its evaluation protocol. Without the original individual predictions, we cannot verify the inferred corrected Bamboogle scores. We will clarify this issue in the manuscript.

Importantly, GiGPO performs below AT²PO at both model scales in our reproduced experiments. Since AT²PO remains our strongest baseline, this GiGPO-specific aggregation issue cannot explain SLPO's advantage over AT²PO and does not materially affect the central comparison supporting our findings.

3. COVER-EM and Evaluation Objectives

The effectiveness of an RL method should primarily be assessed by how well it improves the objective it is trained to optimize. In our experiments, COVER-EM is not an arbitrarily selected post-hoc metric: it is the common final-answer correctness criterion used in training and evaluation across all compared methods.

Under this consistent objective, SLPO achieves the largest improvement among the compared methods at both model scales. This directly demonstrates its effectiveness in optimizing the intended task objective.

We understand that EM and F1 measure other aspects of answer quality. However, applying them retrospectively to checkpoints trained and selected for COVER-EM would introduce a mismatch between the training objective and the additional evaluation criteria. EM and F1 may provide complementary perspectives, but they do not invalidate the demonstrated improvement under the shared COVER-EM objective.

We believe the new evaluation directly addresses the principal concern about checkpoint-selection bias, while the remaining issues do not materially undermine the central empirical findings or the novelty and soundness of SLPO's state-local training formulation. We hope the reviewer will reconsider the assessment in light of this additional evidence.

Reviewer_kdXpOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes State-Local Policy Optimization (SLPO) for training LLM search agents with RL. Instead of giving all actions in a trajectory the same outcome-based advantage, SLPO samples several alternative actions under the same search history, scores each one, and normalizes the rewards within that state. To keep this affordable, it expands only representative queries in a semantic tree, rewards searches that retrieve new evidence using an offline LLM-annotated evidence mapping, and samples training states in proportion to the spread of their action rewards. On seven QA datasets with Qwen3-4B-Instruct and Qwen3-8B, SLPO improves Cover-EM over the strongest baseline (AT2PO) by 3.54 and 3.84 points while using fewer searches than Search-R1 and PiCA.

Critical strengths
  • Well-motivated problem. Trajectory-level advantages give the same credit to useful and redundant searches, and the paper makes a clear case for comparing actions under the same history. The evidence reward is computed from retrieved document IDs, so no LLM judge is needed during training.
Critical weaknesses
  • The experiments do not support the claimed improvement. All results are reported on the same validation mixture used to select checkpoints, with no held-out test set, and each method is reported at its best checkpoint from a single run. The fixed 200-step window also favors SLPO at 4B: SLPO reaches its best score at step 80, while AT2PO, the strongest baseline, gains about 3 points in its last 20 steps (Figure 3). The only seed check (4B, step 100) gives 65.46 ± 0.51, and all three seeds fall below the main run at the same step.

  • The contribution of state-local credit assignment is limited and not isolated. TreeRPO [1] already groups the child steps of a node and GiGPO [2] groups repeated states, while information-gain rewards with redundancy penalties come from StepSearch [3]. The authors acknowledge that local rewards, within-state normalization, response-only updates and state selection "are coupled, so the controls do not isolate their individual contributions" (Section 5). SLPO also uses LLM-annotated evidence that the four trained baselines (Search-R1, PiCA, AT2PO and GiGPO) do not have, and against the Evidence Credit control, which uses the same evidence, the gain shrinks to 2.46 points at 4B. The gains over the main baselines therefore cannot be attributed to state-local credit assignment.

References

[1] Zhicheng Yang, Zhijiang Guo, Yinya Huang, Yongxin Wang, Yiwei Wang, Xiaodan Liang, and Jing Tang. TreeRPO: Tree Relative Policy Optimization. arXiv:2506.05183, 2025.

[2] Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-Group Policy Optimization for LLM Agent Training. NeurIPS 2025.

[3] Xuhui Zheng, Kang An, Ziliang Wang, Yuhang Wang, and Yichao Wu. StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization. EMNLP 2025.

Questions for authors
  • What are the results with checkpoints selected on a held-out split and reported on a separate test set, with seed variance for all methods?
  • How do the baselines perform when trained to convergence, or with the same wall-clock budget as SLPO, which costs 2.10x per step relative to the Evidence Credit control (Table 13)?
  • How does SLPO perform with outcome rewards only?
4 comments
AuthorsOpenReview ↗

Response to Reviewer kdXp

We thank the reviewer for recognizing the problem's importance. Two distinctions matter: the seed check and main table use different statistics, and prior state grouping differs from same-history comparison.

1. Evaluation and Reproducibility

The gains are not due to a SLPO-specific checkpoint rule. We followed AT²PO's published best-checkpoint practice, using the same 0–200-step window, data, and metric for every method. The 200-step budget follows GiGPO's 200 iterations and AT²PO's original curve, which shows little further gain around steps 200–220. Although reproduced AT²PO was still improving, Figure 3 shows SLPO ahead at fixed step 200 at both model scales—not only at its best checkpoint.

The review compares different statistics: 65.46 ± 0.51% was a fixed-step-100 three-seed check to reduce validation cost, while 68.34% used best-checkpoint selection. We re-evaluated three independent runs every 20 steps, selecting each run's best checkpoint: 68.12 ± 0.46%. Its agreement with 68.34% shows the reported performance is repeatable, not an isolated high-scoring run. Repeating every baseline across seeds is expensive in agentic RL; the original Search-R1, AT²PO, and GiGPO QA evaluations likewise do not report multi-seed variance (GiGPO reports three seeds for ALFWorld/WebShop). We report validation results; the original baseline papers likewise lack a separate validation-to-held-out-test protocol.

2. Training Cost and Convergence

Higher per-update cost is not the same as worse training efficiency. SLPO's extra time comes from tree exploration, candidate scoring, and actor computation—not local normalization alone (Table 13). Recorded time to each best 4B result:

MethodCOVER-EMTime
SLPO68.34%2.63 h
Evidence Credit65.88%3.89 h
GiGPO59.40%9.03 h
PiCA61.00%11.13 h
AT²PO64.80%15.84 h

With method-specific throughput settings, these are observed runtimes, not a strict compute-matched test. SLPO reached the highest score fastest. Section 5 discusses pruning and early stopping to reduce tree overhead.

3. Why the Cited Methods Are Different

Sharing elements of step-level credit does not make these methods equivalent. Tool calls create natural decision boundaries. SLPO independently executes alternatives from exactly the same search history, compares their directly observed rewards, and updates only the current action.

TreeRPO divides continuous mathematical reasoning into token-length segments, an imprecise step definition acknowledged by its authors, and derives intermediate values from downstream outcomes. SLPO uses natural tool boundaries and immediate retrieval feedback. Its separate-action representation also hides historical reasoning during both rollout and training while preserving the current action's original tokens, directly addressing the context growth and training–inference mismatch highlighted by veRL.

GiGPO's “same state” is not ours. Its anchor groups identify repeated environment observations without interaction histories. Identical observations can follow different searches with different evidence already covered; they are not identical decision contexts. SLPO deliberately compares actions under one identical history. GiGPO also derives step credit from downstream returns, whereas SLPO scores each candidate search's immediate evidence gain.

StepSearch introduces information-gain and redundancy rewards, but not independent same-history comparisons with within-state relative advantages. SLPO uses verified claim–document mappings for deterministic rewards; semantic clustering limits redundant branch expansion, and batch-level selection prioritizes informative states.

4. Additional Supervision and Outcome-Only Rewards

Extra evidence supervision alone cannot explain the gain. PiCA also uses DeepSeek annotations to train a process reward model, yet obtains 61.00% in our unified 4B comparison. Evidence Credit uses the same mapping as SLPO but scores 65.88%, versus 68.34% for SLPO. We further tested sign reversal of trajectory advantages (no gain over Evidence Credit) and trajectory GRPO using the sum of SLPO's step rewards (worse than original outcome-only GRPO). We will add both controls to the revision.

An outcome-only reward cannot assess a search when it is made: comparing candidate searches by final correctness requires continuing each branch, adding rollouts and mixing their credit with later decisions—precisely what immediate evidence feedback avoids. Outcome-only trajectory GRPO scores 63.91%; adding downstream outcomes via Subtree Reward scores 65.38%, below SLPO's 68.34% (Table 3). The contribution is not simply “trees plus rewards,” but assigning credit to the action that acquired the evidence.

I thank the authors for the rebuttal and clarification, which partially address my concerns on the reproducibility of the main result and the differences from TreeRPO and GiGPO. To reflect this, I have raised my rating to 2. The main reasons that keep me from a higher score are twofold:

  • As acknowledged, all numbers are validation results instead of a held-out evaluation, and checkpoints are selected on the same validation set. Applying the same rule to all methods or following AT²PO does not make this protocol sound: it measures which method benefits most from selection on the validation set rather than the true performance (especially on unseen scenarios) of each method. A sound comparison should select checkpoints on a separate set and evaluate on a held-out set. Given the current setup, the empirical significance cannot be established.

  • Additionally, the insufficient ablation study makes the claimed key contribution of state-local credit assignment unclear. Besides the proposed state-local credit assignment, SLPO also uses additional evidence rewards that the outcome-reward baselines do not have. The control study in Table 3 fails to separate the impact of state-local credit assignment from other SLPO-only components such as response-only updates and state selection, which none of the trajectory-level controls has. Therefore, the real contribution of state-local credit assignment remains unclear.

Given these two remaining issues, I am not convinced of the claimed effectiveness of the method.

AuthorsOpenReview ↗

Follow-up Response to Reviewer kdXp

We thank the reviewer for reconsidering our work and raising the rating. We appreciate the remaining concerns and would like to clarify two points.

1. Evaluation and Generalization

We believe the conclusion that our empirical results cannot establish the method's effectiveness is too strong. As detailed in our Response to Common Reviewer Concerns, the original Search-R1, GiGPO, PiCA, and AT²PO papers likewise do not report a separate held-out evaluation after model selection.

Regarding performance on unseen scenarios, SLPO is trained only on the HotpotQA training split but evaluated across seven QA benchmarks, including six other datasets not used for RL training. These results provide cross-dataset evidence beyond the training data, covering both single-hop and multi-hop tasks.

Furthermore, Figure 3 shows SLPO outperforming the baselines at corresponding evaluation steps throughout training, including fixed step 200. Its advantage therefore does not depend solely on selecting the highest validation score.

A separate held-out test would provide a stronger estimate of generalization, but the existing cross-dataset results and consistent learning curves still provide meaningful empirical evidence for SLPO's effectiveness.

2. Component Attribution and the Complete Method

Our contribution is the state-local training formulation as a whole, not within-state normalization in isolation. Same-history action comparison, response-only updates, and state selection are deliberately designed to work together, aligning rewards, relative advantages, and policy updates at the level of individual decisions rather than entire trajectories.

The evidence-grounded reward design is also a contribution of this work, providing interpretable and reproducible process feedback without training a separate reward model. External supervision is not unique to SLPO: PiCA (NeurIPS 2026) also uses DeepSeek annotations to train a process reward model. In our unified evaluation, SLPO outperforms both PiCA and AT²PO.

We have also conducted two additional trajectory-level controls to further investigate performance attribution. Together with Evidence Credit, these experiments show that incorporating process supervision into trajectory-level optimization does not reproduce SLPO's gains, as detailed in our earlier responses.

The need to further isolate the contributions of interacting components should not overshadow the demonstrated effectiveness and value of the complete method. With aligned learning rates, 4B SLPO surpasses AT²PO's best score in our comparison by step 80, and 8B does so by step 60, while AT²PO reaches its best at step 200. Such faster learning in terms of training updates is itself a meaningful advantage.

We believe these results, together with the proposed training formulation and evidence-grounded reward design, provide substantial support for the effectiveness and overall contribution of this work.

AuthorsOpenReview ↗

Follow-up Response to Reviewer kdXp

To directly address the remaining concern about checkpoint-selection bias, we reselected checkpoints using only HotpotQA validation and evaluated them on the other six QA datasets. All methods were trained on the HotpotQA training split.

ModelSLPOAT²POImprovement
4B68.65%64.64%+4.01 pp
8B66.98%63.13%+3.85 pp

SLPO retains substantial improvements at both model scales when checkpoint selection and evaluation use separate datasets. Consistent with our original findings, these results show that the advantage is not merely an artifact of selecting checkpoints on the evaluation mixture, while providing further evidence of cross-dataset generalization.

We hope this additional analysis addresses the reviewer's concern about the reliability of our empirical results.

Reviewer_dpbEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes State-Local Policy Optimization (SLPO) for training LLM-based search agents. Instead of assigning a single trajectory-level advantage to all actions, SLPO samples alternative actions under the same search history, scores them using new evidence acquisition or answer correctness, and applies within-state relative advantages to the corresponding response tokens. Semantic query clustering limits tree expansion while retaining sampled actions for local scoring. Reward dispersion determines group eligibility and selection, with cyclic reuse when too few informative groups are available.

Experiments on seven QA validation datasets report overall COVER-EM scores of 68.34% and 66.30% for the 4B and 8B models, respectively, exceeding the strongest reported baseline at both scales. The paper also examines evidence-based credit assignment, search behavior, and training efficiency.

Critical strengths

S1. Clear motivation for local credit assignment.

Useful searches can occur in unsuccessful trajectories, while successful trajectories can contain redundant searches. Comparing alternative actions under the same history is a sensible approach to this credit-assignment problem. Independently executing non-medoid actions also avoids assuming that semantically similar queries necessarily retrieve the same evidence.

S2. Encouraging results across multiple datasets and model configurations.

The evaluation covers seven QA datasets and two model configurations, with improvements in both single-hop and multi-hop aggregate accuracy. SLPO also uses fewer search calls than Search-R1 and PiCA. The evidence-credit trajectory control provides useful evidence that intermediate retrieval information can improve training.

S3. Detailed reporting of implementation and practical tradeoffs.

The appendices describe the reward schedule, action parsing, serialization, evidence mapping, and baseline settings. The authors also report higher per-update cost, substantial group reuse, and a high fraction of uninformative candidates. These details help readers assess the implementation and its practical limitations.

Critical weaknesses

W1. Insufficient isolation of the proposed source of improvement.

The Evidence Credit baseline in Appendix B.5 preserves outcome-derived trajectory advantages and only rescales search-token advantages by 0.7 or 1.3. SLPO instead introduces action-specific immediate rewards while also changing grouping, representation, and state selection. Consequently, the 2.46-point gain over this baseline in Table 3 cannot be attributed specifically to same-state comparison. Although Section 5 acknowledges this coupling, a control that holds rewards, response-only representation, and rollout budget fixed while varying the grouping rule is needed to better establish the distinctive contribution.

W2. Evaluation is coupled to checkpoint selection, with limited uncertainty estimates for the main comparison.

Section 4.1 uses the same 51,713-example validation mixture to select the best checkpoint over steps 0–200 and to report performance. Although this selection rule is shared across methods, the selected maxima do not provide an independent estimate of generalization. The three-seed result of 65.46±0.51% at step 100 is useful, but it does not quantify variability for the 68.34% best-checkpoint result or establish a matched multi-seed advantage over AT2PO. Evaluation on data excluded from checkpoint selection, together with matched repeated runs, would strengthen the main empirical claim.

W3. The validity of the evidence reward needs more direct evaluation.

The annotation and verification pipeline is described, but no independent audit of mapping errors or missed alternative evidence is reported. Appendix A.1 also truncates retrieved observations and removes older history, while Appendix A.4 defines rewards using document identifiers and accumulated evidence coverage. This creates a potential mismatch between rewarded evidence and the supporting text actually visible to the model. Previously retrieved evidence that is no longer visible might also receive a redundancy penalty when retrieved again. Measuring these cases and clarifying how C(s) interacts with history truncation would help establish the reliability of the reward signal.

W4. A reported baseline result is inconsistent with the stated scoring protocol.

Table 8 lists 125 Bamboogle examples, Table 10 specifies greedy decoding with n=1, and Appendix B.1 defines accuracy as an example-level binary mean. Under these conditions, accuracy must change in increments of 0.8 percentage points. The GiGPO scores of 23.39% and 73.79% in Table 2 cannot arise from that protocol. A different denominator or an additional averaging procedure could explain the discrepancy, but the authors should provide the correct/total counts and clarify any exclusions or aggregation across batches, workers, or runs.

Questions for authors

NA

1 comment
AuthorsOpenReview ↗

Response to Reviewer dpbE

We thank the reviewer for the encouraging assessment and specific feedback.

W1. What Drives the Improvement?

EC was designed to test evidence feedback within trajectory-level optimization, not to replicate SLPO. Following sign-preserving credit adjustments in STAMP, CW-GRPO, and TAPO, it retains outcome-derived advantage signs. To address the sign concern, we also tested reversing selected signs using evidence feedback, with no clear gain over EC. In another control, we summed SLPO's step rewards into a trajectory reward and applied standard trajectory-level GRPO; it performed worse than outcome-only GRPO. We will add both experiments to the revision.

Additional process supervision is not unique to SLPO: PiCA trains a step-reward model using DeepSeek-annotated pivot steps yet reaches 61.00% in our unified 4B evaluation. EC reaches 65.88%, while SLPO reaches 68.34% (Table 3). SLPO's distinction is to execute alternatives under exactly the same search history, compare immediate rewards within that state, and update only the corresponding action tokens. This also allows historical reasoning to be hidden during both rollout and training, addressing the context-growth and training–inference mismatch identified in veRL's multi-turn documentation.

Adaptive semantic-tree branching makes identical rollout counts difficult to enforce. We therefore report observed training time to best 4B scores: SLPO 2.63 h (68.34%), EC 3.89 h (65.88%), and AT²PO 15.84 h (64.80%). We aligned optimization settings where appropriate but retained method-specific throughput configurations, so these are not compute-matched times.

W2. Checkpoint Selection and Variability

We followed AT²PO's published best-checkpoint practice, applying the same 0–200-step selection rule, data, and metric to all methods. A separate validation-selection/held-out-test protocol is not standard across the search-agent RL baselines considered here. More importantly, Figure 3 shows SLPO ahead at fixed step 200 at both model scales: the advantage does not depend on choosing an isolated peak.

The review compares different statistics. The original 65.46 ± 0.51% was measured at fixed step 100 to limit validation cost; the main 68.34% was selected across checkpoints. To make the comparison consistent, we evaluated three independent runs every 20 steps and selected each run's best evaluated checkpoint, obtaining 68.12 ± 0.46%. Its close agreement with 68.34% shows that the main validation result is repeatable, rather than an exceptional run. These remain validation results.

W3. Evidence Reliability and Visible History

The annotation procedure specifically targets incorrect and missing evidence, rather than trusting an LLM's first output. We give the annotator both the question and correct answer, reducing the task to finding support instead of solving the question. A context-isolated verifier checks individual claims and whether they jointly establish the answer; failed attempts are revised or excluded. We then expand each claim's supporting documents through two retrievals and independently verify the additional matches (Appendix A.3). Multiple documents supporting one claim cannot earn repeated positive rewards. These checks improve reliability and coverage, though they do not imply an exhaustive mapping.

On truncation, C(s) intentionally tracks evidence acquired over the full interaction, preventing repeated retrieval of an already found fact from earning fresh credit. Prompt lengths suggest limited context pressure: across 200 updates, mean prompt length averaged 744.26 tokens; per-update maximum length averaged 2,277.67, with a highest observed value of 3,954 against the 4,096-token rolling limit. Hiding earlier reasoning in the training representation also reduces context use, which may help explain these lengths. These statistics do not directly measure dropped history or whether the 1,024-token observation cap hides supporting passages. We will clarify the distinction between accumulated evidence and visible context.

W4. Bamboogle Aggregation

Our GiGPO reproduction inherited unweighted averaging of validation-batch accuracies from its public verl-agent implementation, as independently reported in Issue #267. A 124+1 split of 125 Bamboogle examples reproduces the reported 23.39% and 73.79%; the corresponding example-weighted scores would be 46.40% and 48.00% if the inferred counts are confirmed. Without the original predictions, we do not present these as verified corrections. Bamboogle accounts for only 125 of 51,713 examples, so this specific anomaly cannot explain the overall performance gap. We will clarify the aggregation and affected results.

Reviewer_jh5oOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SLPO, a state-local credit assignment method for training search agents. At each search history, it samples several candidate actions and scores each one with an immediate evidence-based reward built offline from gold answers. It then normalizes the rewards among candidates at the same state and updates only the current action. Trained on HotpotQA, SLPO outperforms prior methods on seven QA benchmarks at two model scales.

Critical strengths
  1. The motivation is clear. Comparing actions under the same history is a sensible way to address trajectory-level credit assignment.
  2. No critic or LLM judge is needed during training.
Critical weaknesses
  1. The ablation does not isolate the state-local component. The Evidence Credit control keeps the sign of the trajectory advantage, so useful searches in failed trajectories are still penalized. Because SLPO also changes the reward and the state selection, the gain may come from the extra gold-derived supervision rather than from state-local comparison. The same concern applies to the comparison with AT²PO. A control that uses the same per-action rewards with trajectory-level normalization is needed.
  2. The evaluation may overstate the gains. Checkpoints are selected and reported on the same validation set, and the fixed-step multi-seed results are clearly lower than the main table. Results on a held-out set or at a fixed step are needed.
Questions for authors
  1. The immediate reward only measures what a single step retrieves, and all benchmarks here are short-horizon. In longer tasks, where a search may pay off only several steps later, does this myopic signal still work? Results on a harder benchmark such as BrowseComp-Plus would help.
  2. The reward relies on an evidence mapping built offline with an LLM from gold answers. How sensitive is SLPO to the quality and coverage of this mapping? For example, how would performance change with a noisier or sparser mapping, or with a reward that does not require gold answers?
2 comments
AuthorsOpenReview ↗

Response to Reviewer jh5o

We thank the reviewer for recognizing SLPO's motivation and for the constructive suggestions.

W1. State-Local Credit vs. Evidence Supervision

We have now tested both concerns directly and will add these results to the revised paper. Evidence Credit (EC) deliberately preserves trajectory-advantage signs, following sign-preserving process-credit methods such as STAMP and CW-GRPO. This tests evidence weighting without changing trajectory-level credit direction. We ran two further controls:

  1. Sign reversal: We allowed evidence feedback to reverse the advantage sign for selected actions. This did not improve on EC, so the sign constraint alone does not explain the gap.
  2. Trajectory GRPO with the same transition rewards: We summed the rewards that SLPO would assign to each step into a single trajectory reward, then applied standard trajectory-level GRPO normalization. This performed worse than the original outcome-only Trajectory GRPO.

Additional process supervision is not unique to SLPO: PiCA also uses DeepSeek annotations to train a process reward model, yet achieves 61.00% in our unified 4B evaluation. Neither richer rewards nor more aggressive trajectory credit reproduces SLPO's 68.34%, versus 65.88% for EC. SLPO instead evaluates alternative actions under exactly the same search history, normalizes their immediate rewards locally, and updates only the corresponding action tokens. Separate transitions also allow historical reasoning to be hidden consistently during rollout and training, avoiding the multi-turn mismatch noted by veRL.

W2. Checkpoint Selection and Reproducibility

Our reported best checkpoint is not a SLPO-specific evaluation choice. We followed AT²PO's published best-checkpoint practice and applied the same 0–200-step rule, dataset, and metric to every baseline. A separate validation-selection/held-out-test protocol is not uniformly used in these search-agent RL studies. Furthermore, Figure 3 shows SLPO ahead on both model scales even at the fixed 200-step checkpoint, so its advantage does not hinge on selecting a favorable peak.

The reviewer compares two different statistics: 68.34% is selected across checkpoints, while the earlier 65.46 ± 0.51% was deliberately measured at fixed step 100 to limit validation cost. To address this comparison, we evaluated three independent runs every 20 steps and selected each run's best evaluated checkpoint. The result is 68.12 ± 0.46%, closely matching 68.34%. The main result is thus repeatable across runs, not an isolated high-scoring checkpoint. These are explicitly validation results.

Q1. Longer-Horizon Search and BrowseComp-Plus

Our seven benchmarks are standard in search-agent RL and cover both single-hop and multi-hop QA. BrowseComp-Plus tests substantially harder deep search: its reported Search-R1-32B accuracy with BM25 is only 3.86%, making it a different setting from our 4B/8B models with a five-search budget.

The guiding idea is to understand and decompose a task before designing local rewards. In QA, we identify the supporting facts needed to connect a question to its answer, including necessary intermediate facts—not just the final answer itself. Retrieving such a fact can therefore receive immediate credit, even when the agent cannot yet answer the question. This avoids relying on the outcome of later actions to evaluate each search. Our Subtree Reward variant incorporates downstream outcomes but did not improve performance (65.38% vs. 68.34%, Table 3). Extending this task-driven decomposition to longer, less directly verifiable searches such as BrowseComp-Plus is a valuable next direction.

Q2. Evidence Mapping Quality and Gold Answers

We designed the annotation pipeline to reduce labeling difficulty and mapping errors. Because the annotator receives both the question and correct answer, it needs to identify supporting facts rather than solve the question from scratch. A separate, context-isolated verifier checks factual support and whether the claims jointly establish the answer; failed annotations are retried or excluded. We also expand supporting documents for each verified claim and independently check whether they support that claim (Appendix A.3). These steps target incorrect and missing mappings rather than treating an LLM's initial output as ground truth.

Systematic noisy/sparse-mapping sweeps would require additional RL training and are not standard in the reported evaluations of outcome-supervised baselines such as Search-R1 and AT²PO. SLPO itself does not require gold answers: state-local optimization needs verifiable transition rewards, and gold-conditioned evidence mapping is our implementation for QA.

Thank you for the additional analyses. The revised checkpoint-selection protocol and multi-seed results substantially alleviate my evaluation concerns, and the additional controls help address the source of the gains. On balance, I am raising my rating to Weak Accept.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Common Reviewer Concerns

We thank all reviewers for their thoughtful feedback. Since several concerns recur across the reviews, we clarify two central issues: our evaluation protocol and the overall contribution and value of SLPO. Detailed responses are provided in our reviewer-specific replies.

1. Evaluation Protocol

The 0–200-step window was not tailored to SLPO. The original training settings of our compared baselines provide clear precedents:

  • Search-R1: 500 steps; its reported GRPO runs frequently collapse around steps 160–200.
  • PiCA: 190 steps.
  • GiGPO: 200 steps for search QA.
  • AT²PO: 240 steps; its published curve extends to step 220 and shows limited further improvement near the end.

For checkpoint selection, we follow our strongest baseline, AT²PO, which reports its best evaluation checkpoint, applying the same 0–200-step window, data, metric, and selection rule to all reproduced methods. Search-R1 and GiGPO instead report final or last valid checkpoints. The original baseline papers also do not report a separate held-out evaluation after model selection.

Figure 3 shows SLPO ahead at corresponding evaluation steps throughout the plotted curves at both model scales. Its advantage holds under both fixed-step comparisons and best-checkpoint reporting, rather than depending on a favorable selection rule.

2. Overall Contribution and Value

SLPO outperforms both AT²PO (ACL 2026), which does not use additional process annotations, and PiCA (NeurIPS 2026), which trains a reward model using DeepSeek annotations. Our evidence-grounded rewards improve trajectory-level training through Evidence Credit, while full SLPO achieves further gains. We also conducted two additional controls to investigate performance attribution, as detailed in our individual responses.

Our work introduces a new perspective on training multi-turn tool-using agents and develops an interpretable, reproducible evidence-grounded reward design for QA. SLPO compares actions under the same history at natural tool-interaction boundaries and optimizes them directly. The QA reward pipeline provides verifiable process feedback without training an additional reward model.

The additional controls further strengthen our analysis. Difficulty in perfectly isolating interacting components should not diminish the demonstrated effectiveness and value of the complete method. With aligned learning rates, 4B SLPO surpasses AT²PO's best score in our comparison by step 80, and 8B does so by step 60, while AT²PO reaches its best at step 200. These results demonstrate stronger performance and faster learning, supporting the overall contribution and value of this work.

Dear Reviewers,

Following up on the authors' message and my previous reminder about responding to them:

Let me remind you that the deadline of the discussion period is Nov. 18 so please please read the reviewers' response and take a minute to write an acknowledgment and confirm your final evaluation, as soon as possible.

Thanks a lot for your help.

AuthorsOpenReview ↗

Follow-up on Common Reviewer Concerns

To directly address the concerns about checkpoint-selection bias, we conducted an additional analysis. All methods were trained on the HotpotQA training split. We reselected checkpoints using only HotpotQA validation and evaluated them on the other six QA datasets, which were excluded from checkpoint selection. We also used AT²PO's learning rate of 1e-6 without tuning SLPO's optimization hyperparameters against the validation mixture.

ModelSLPOAT²POImprovement
4B68.65%64.64%+4.01 pp
8B66.98%63.13%+3.85 pp

SLPO maintains substantial improvements at both model scales even when checkpoint selection and evaluation use separate datasets. Consistent with our original findings, these results show that the advantage is not merely an artifact of checkpoint selection on the evaluation mixture, while providing further evidence of cross-dataset generalization.

We hope this additional analysis helps address the shared concerns about the reliability of our empirical results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.