- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses credit assignment over long trajectories in RL for search agents, and proposes SLPO. Through semantic-tree exploration, SLPO samples multiple candidate actions under the same search history. It then uses an offline-constructed evidence-to-document mapping to assign immediate transition rewards. Search actions are rewarded for newly acquired evidence. Answer actions are rewarded for a combination of correctness and evidence coverage. Experiments use two Qwen3 models of different scales and a validation mixture of seven QA datasets. The results show that SLPO improves COVER-EM over the strongest baseline and uses fewer search calls than some baselines.
Critical strengths
- The problem is practical and significant. RL with outcome rewards cannot distinguish the values of individual step-level actions, which is a long-standing problem in the field.
- Transparency and reproducibility are good. The paper provides extensive experimental details, which makes reviewing easier.
Critical weaknesses
- No independent final evaluation. The authors select and report the best checkpoint on the same evaluation mixture, so the reported scores may carry an optimistic bias from model selection. Applying the same procedure to all methods does not guarantee that this bias is equal across methods. This may distort the comparison with the baselines.
- Incompleteness. The paper still appears insufficiently complete, and the main text is too thin.
- Lenient primary metric. The evaluation mainly uses COVER-EM, which is far more lenient than EM and F1. It also carries the risk of inflating ("hacking") the score by outputting more content.
- Questionable results. Bamboogle has only 125 examples, so every score on it should be a multiple of 0.8%. GiGPO's scores of 23.39 (4B) and 73.79 (8B) are not, and it is unclear whether examples are missing or there is some other issue.
- Unequal privileged information. The method uses DeepSeek to construct additional privileged information, but most of the compared baselines do not have access to it. This makes the comparison insufficiently fair.
- Incomplete evidence mapping. The Wiki corpus may contain many documents supporting the same fact. A limited number of fixed retrievals can hardly exhaust all valid evidence documents. As a result, a valid search may be wrongly penalized because it misses the offline mapping.
- No semantic deduplication. The authors decide whether information is new based on atomic facts (evidence items) and their corresponding document IDs. However, they do not explicitly design a deduplication mechanism that accounts for semantic equivalence or entailment across evidence items. If two semantically equivalent facts are annotated as different evidence items, a repeated retrieval may be wrongly rewarded.
- Myopic rewards. The reward design considers only the gain from a single-step transition. For necessary exploratory queries, which cannot directly provide the evidence needed to answer the question, the framework cannot assign credit correctly.
- Heuristic design. The constants in the reward schedules for the two action types come with no explanation of how they were chosen. Neither the specific constants nor the relative scale between search rewards and answer rewards is adequately justified or supported by a sensitivity analysis.
Questions for authors
See Weaknesses.
3 comments
Thank you for the response. It clarified some of my concerns, but three issues remain. First, I understand that prior work has used the same evaluation protocol. However, this precedent does not resolve the potential selection bias introduced by selecting checkpoints and reporting performance on the same validation set. I would still like to see an evaluation on an independent held-out set.
Second, the authors offer a plausible explanation for the Bamboogle results. However, giving equal weight to a batch containing 124 Bamboogle examples and another containing just one is inconsistent with the per-example accuracy described in the paper. The response does not provide scores verified against the individual predictions.
Finally, the existing predictions could be rescored using exact match and token-level F1 to assess whether the main findings are robust to the choice of metric. This analysis requires no retraining, but the response does not report it. Given these unresolved concerns, I am maintaining my original rating.
Further Response to Reviewer bP4g
We thank the reviewer for the follow-up. We have reanalyzed our existing evaluation results to directly address the checkpoint-selection concern and would like to clarify the remaining issues.
1. Independent Evaluation
Both SLPO and AT²PO are trained on the HotpotQA training split. We now select checkpoints using only HotpotQA validation and report performance on the other six QA datasets, which are excluded from checkpoint selection.
The results are:
| Model | SLPO | AT²PO | Improvement |
|---|---|---|---|
| 4B | 68.65% | 64.64% | +4.01 pp |
| 8B | 66.98% | 63.13% | +3.85 pp |
SLPO maintains substantial improvements at both model scales when checkpoint selection and final evaluation use separate datasets. The gains are consistent with our original findings, directly addressing the concern that SLPO benefits disproportionately from selecting checkpoints on the same evaluation mixture.
These results also demonstrate that SLPO's advantages extend to six QA benchmarks not used for RL training or checkpoint selection under this protocol.
2. GiGPO Aggregation
We agree that unweighted batch averaging differs from the stated per-example accuracy. This issue originates from the publicly released GiGPO implementation used in our reproduction, rather than a deliberate modification of its evaluation protocol. Without the original individual predictions, we cannot verify the inferred corrected Bamboogle scores. We will clarify this issue in the manuscript.
Importantly, GiGPO performs below AT²PO at both model scales in our reproduced experiments. Since AT²PO remains our strongest baseline, this GiGPO-specific aggregation issue cannot explain SLPO's advantage over AT²PO and does not materially affect the central comparison supporting our findings.
3. COVER-EM and Evaluation Objectives
The effectiveness of an RL method should primarily be assessed by how well it improves the objective it is trained to optimize. In our experiments, COVER-EM is not an arbitrarily selected post-hoc metric: it is the common final-answer correctness criterion used in training and evaluation across all compared methods.
Under this consistent objective, SLPO achieves the largest improvement among the compared methods at both model scales. This directly demonstrates its effectiveness in optimizing the intended task objective.
We understand that EM and F1 measure other aspects of answer quality. However, applying them retrospectively to checkpoints trained and selected for COVER-EM would introduce a mismatch between the training objective and the additional evaluation criteria. EM and F1 may provide complementary perspectives, but they do not invalidate the demonstrated improvement under the shared COVER-EM objective.
We believe the new evaluation directly addresses the principal concern about checkpoint-selection bias, while the remaining issues do not materially undermine the central empirical findings or the novelty and soundness of SLPO's state-local training formulation. We hope the reviewer will reconsider the assessment in light of this additional evidence.
Follow-up Response to Reviewer bP4g
We appreciate the reviewer's continued consideration. Our earlier response addressed the nine concerns individually; here we clarify the central distinction behind our contribution and provide additional experimental evidence.
1. The Core SLPO Formulation vs. Its QA Reward Implementation
Most concerns address QA reward construction or evaluation, rather than state-local optimization itself. SLPO changes how multi-turn agents learn: each tool interaction provides a natural decision boundary. It independently executes alternatives under exactly the same search history, computes within-state advantages from observed rewards, and updates only the corresponding action tokens.
This representation also permits historical reasoning to be hidden during rollout and training, addressing context growth and training–inference mismatch identified in veRL's multi-turn documentation.
The evidence mapping provides verifiable QA rewards, not a mathematical requirement of state-local optimization. Concerns about its completeness should be distinguished from the soundness and originality of the training formulation.
2. The Evaluation Protocol Does Not Favor SLPO (W1)
We did not introduce the best-checkpoint rule to favor SLPO. It follows our strongest baseline, AT²PO, which explicitly reports the checkpoint with the highest average evaluation score (Sec. 5.1). The original Search-R1, GiGPO, PiCA, and AT²PO papers likewise do not report a separate held-out evaluation after checkpoint selection. We explicitly label our scores as validation results.
The 200-step window was chosen based on prior practice: GiGPO uses 200 iterations for QA, and AT²PO's published curve shows little further improvement around steps 200–220. We apply the same window, data, and metric to all reproduced methods. Figure 3 shows SLPO ahead even at fixed step 200 at both model scales, so its advantage does not depend on selecting a favorable peak.
3. Evidence Supervision Alone Does Not Explain the Gain (W5)
SLPO is not the only method using external process supervision. PiCA uses DeepSeek annotations to train a step-reward model, yet scores 61.00% in our unified 4B evaluation. Our Evidence Credit control uses the same mapping as SLPO but retains trajectory-derived advantages: 65.88% versus 68.34% for SLPO.
We also tested (i) reversing selected advantage signs using evidence feedback, which did not improve on Evidence Credit, and (ii) summing SLPO's step rewards into a trajectory reward for standard GRPO, which performed worse than outcome-only GRPO. Both results will be added to the revision. These controls show that richer supervision or trajectory-level reward changes alone do not reproduce SLPO's improvement.
4. Task-Driven Rewards and Intermediate Evidence (W6–W9)
Characterizing our reward as “myopic” overlooks its deliberate task-driven design. The guiding idea is to understand and decompose a task before designing local rewards. In search QA, successful answering requires supporting facts linking the question to its answer. We can therefore assess meaningful intermediate progress without waiting for the final outcome.
Given both question and correct answer, our annotator identifies atomic supporting claims rather than solving the question. An independent verifier checks that the claims jointly support the answer. These include necessary intermediate facts, not merely statements of the final answer. Retrieving them earns immediate credit even before the agent can answer. This is intentional: it recognizes useful actions without introducing uncertainty from later decisions.
Each claim also maps to multiple independently verified documents, but earns reward only upon first coverage (Appendix A.3, Eq. 6). Different documents supporting the same mapped fact cannot generate repeated rewards. This handles document-level redundancy, though exhaustive coverage and cross-claim deduplication are not guaranteed.
Some exploratory actions may have delayed benefits beyond mapped evidence. Adding downstream outcomes via Subtree Reward did not improve performance (65.38% vs. 68.34%, Table 3). Extending task-driven rewards to less directly verifiable settings remains worthwhile.
We believe these QA-specific concerns (addressed in our earlier response and further clarified above) do not materially undermine our main findings or establish a flaw in SLPO's state-local optimization. Our central contribution is to assign credit to individual decisions under the same history, rather than letting later trajectory outcomes obscure their contributions. The additional controls and experimental results support the value of this formulation beyond merely introducing richer reward annotations.