acceptodds
Under review as a conference paper at ICLR 2027

What Does Answer-Based Co-Training Teach LLM Search Agents?

Abstract

Agentic search enables large language models to solve knowledge-intensive tasks through repeated retrieval and reasoning. Reinforcement learning trains these agents with final-answer rewards, but sparse feedback gives limited guidance for intermediate search decisions. Answer-based co-training addresses this problem by training a model to answer at intermediate steps and using improvements in answer quality as process rewards. However, how to combine these two training signals into a stronger search agent remains unclear. We investigate four design questions through controlled comparisons with full co-training. First, for parameter sharing, we compare shared and separate models: sharing parameters improves the deployed policy's answers on the same evidence by applying intermediate-answer updates to the policy itself. Second, for training signals, we separate intermediate-answer training from process rewards: intermediate-answer training improves early responses and robustness to changed answering instructions. Third, for reward placement, we compare stepwise delivery with terminal aggregation: accumulating process rewards at the trajectory end provides an effective alternative while preserving their sum. Fourth, for feedback information, we compare agent performance after training with or without prior reasoning in the answer model's input. We separately compare scoring criteria: continued-search scoring identifies useful clues better than immediate-answer scoring by testing whether the policy can follow them to the answer.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM Agentic Search, Reinforcement Learning, Answer-Based Co-Training

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_k46MOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies answer-based co-training for LLM search agents through comparisons of parameter sharing, intermediate-answer training, process-reward placement, and feedback information. It evaluates several training variants on QA benchmarks and uses controlled answering and search diagnostics to examine their behavior.

Critical strengths

The paper considers a reasonably comprehensive set of co-training variants and conducts extensive ablations and diagnostic experiments. These comparisons help separate design choices that are often combined, particularly intermediate-answer training and process rewards.

Critical weaknesses
  1. The paper lacks a clear central message and sufficiently deep analysis. The presentation feels fragmented, with many observations but limited explanation of what they collectively establish. Some findings are relatively expected, such as intermediate-answer training improving performance when the model is asked to answer early. Other results require more analysis: Full underperforms Aux-only on 2Wiki and Reasoning-masked on MuSiQue, but these differences are not adequately explained. Likewise, terminal aggregation performs well in the reported experiment, but the results provide limited guidance on when or why it should be preferred. Overall, the paper reads more like a preliminary collection of experimental observations than a focused study yielding substantial new understanding.

  2. The evidence is insufficiently conclusive to support strong design recommendations. The macro-F1 differences between Full and Aux-only, and between Aux-only and Outcome-only, have 95% confidence intervals that include zero. Several diagnostic evaluations also use small question subsets, and each primary configuration has only one training run. These limitations leave uncertainty about the reliability and generality of the findings. The paper also lacks an explicit analysis separating single-hop and multi-hop QA; additional single-hop benchmarks such as TriviaQA and PopQA would help assess whether the conclusions depend on question type.

Questions for authors
  1. Clarification: What explains the differing configuration rankings across datasets, particularly Full versus Aux-only on 2Wiki and Full versus Reasoning-masked on MuSiQue?

  2. Clarification: Given that the process-reward sum telescopes to the final-state score minus the initial-state score, what substantive insight does the terminal-aggregation experiment provide beyond showing that this alternative reward placement works in the reported setting? What evidence supports a preference between the two placements?

  3. Additional evidence: Do the main conclusions remain consistent with larger diagnostic samples and repeated training runs? An analysis separating single-hop and multi-hop questions would also help establish their scope.

Reviewer_a5nKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents an empirical analysis of answer-based co-training for reinforcement learning of LLM search agents, building upon the OASES framework. The authors investigate four design questions: (1) shared versus separate parameters for search and intermediate answering, (2) the respective contributions of auxiliary answer training and process rewards, (3) stepwise versus terminal process-reward placement, and (4) the information and scoring criteria used to evaluate intermediate search states.

Experiments are conducted using Qwen2.5-7B-Instruct trained with PPO on HotpotQA and 2WikiMultihopQA, with evaluation across five QA benchmarks and additional controlled diagnostic tasks. The results suggest that parameter sharing improves the deployed policy's answering behavior, auxiliary answer training improves early responses and robustness to instruction changes, terminal reward aggregation offers a viable alternative to stepwise placement, and continuation-based scoring can better identify evidence useful for subsequent search.

Critical strengths
  1. Careful analysis through controlled experiments. The paper examines several design choices in answer-based co-training through a series of controlled experiments. Testing different models with the same retrieved contexts makes it easier to determine whether performance differences come from answer generation or evidence retrieval. The comparison between shared and separate parameters also helps clarify how training on intermediate answers influences the search policy used at inference time.

  2. A useful distinction between answering questions and making search progress. The paper shows that evidence can help guide the next retrieval step without necessarily improving the current answer. The synthetic search experiments illustrate this distinction and help explain how intermediate clues affect retrieval and answer generation. This is a useful observation for understanding the limitations of answer-based supervision in multi-step search tasks.

Critical weaknesses

1. Limited novelty and incremental contribution beyond existing work.

The paper substantially overlaps with OASES [1], which already investigates joint optimization, process rewards, and intermediate-state evaluation through ablation studies. The telescoping reward formulation in RQ3 was also derived in OASES. While the present work provides more detailed diagnostics, RQ1 and RQ2 primarily extend existing analyses, and RQ3 offers limited new theoretical insight. RQ4 is potentially more interesting, but future-success-based process supervision has already been explored in Rewarding Progress [2], and this paper evaluates continuation-based feedback only through offline selection rather than actual RL training. Overall, the incremental contribution and practical significance remain insufficiently established.

2. Insufficient experimental robustness and generalizability.

The end-to-end improvements are modest, with Full achieving 50.49 macro F1 compared to 49.45 for Outcome-only. The 95% confidence intervals for both Aux-only versus Outcome-only and Aux-only versus Full include zero. This makes it difficult to establish the individual contributions of auxiliary answer training and process rewards to final task performance. Each configuration is also trained only once, leaving the stability of these findings across random seeds unclear. Notably, extending Outcome-only training yields 51.17 macro F1, surpassing all primary configurations evaluated at the earlier checkpoint, although the comparison is not compute-matched.

The experiments are further limited to Qwen2.5-7B-Instruct and five QA benchmarks introduced between 2018 and 2022. There is no evaluation across model scales or families. Testing on more recent search benchmarks such as FanOutQA [3] and BrowseComp-Plus [4] would help assess whether the findings generalize to more challenging search tasks. Cross-model and multi-seed experiments would also provide stronger evidence for the paper's conclusions.

References

[1] Zhang et al. (2026). OASES: Outcome-Aligned Search-Evaluation Co-Training for Agentic Search. https://arxiv.org/abs/2604.03675

[2] Setlur et al. (2025). Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning. https://arxiv.org/abs/2410.08146

[3] Zhu et al. (2024). FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models. https://aclanthology.org/2024.acl-short.2/

[4] Chen et al. (2025). BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent. https://arxiv.org/abs/2508.06600

Questions for authors
  1. Given that OASES already studies joint optimization, process rewards, and intermediate-state evaluation, what new insights or practical design principles does this paper establish beyond the original work?

  2. Could the authors compare Full, Aux-only, and Outcome-only at later training checkpoints, including 5,035 updates, to determine whether the benefits of co-training persist with longer training?

  3. Could the authors evaluate the main training configurations on additional backbones, such as Qwen2.5-3B and Qwen3-8B, and on more challenging search benchmarks such as FanOutQA or BrowseComp-Plus? How do the improvements vary with search depth?

  4. Given that Aux-only and Outcome-only achieve nearly identical policy-prompt F1 at the last retrieval stage (Table 13), while their early-stage usable-answer rates differ substantially, how much of the reported benefit comes from improved answer generation rather than instruction compliance?

  5. Could the authors train a search policy using continuation-based process rewards and compare its final task performance against the original immediate-answer-based reward? This would help determine whether the RQ4 findings translate into practical improvements in RL training.

Reviewer_jcuHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies OASES-style answer-based co-training, examining parameter sharing, intermediate-answer training, reward placement, and immediate-answer versus continued-search feedback. Its contribution is primarily a controlled empirical analysis based on the factors above.

Critical strengths

The research questions being studied are structured and clear. The four research questions organize the study around reasonable design choices. The analysis and reporting are detailed. The paper distinguishes answer production from content quality, reports paired bootstrap intervals, and uses independent rollout samples for query selection and evaluation.

Critical weaknesses

My major concern is the limited novelty and contribution of this work. It mostly conducts analysis on the existing method OASES, which is more like an extended ablation study and analysis of the existing work. The incremental contribution beyond OASES is limited, including overlap in the ablations themselves. The original OASES paper already studies removing joint optimization, removing process rewards, and removing both. OASES also discusses how search-only training can impair answering from intermediate states. Thus, separating these training signals and identifying intermediate-answer benefits are not entirely new contributions.

The evidence is too narrow to support general design recommendations. All configurations use Qwen2.5-7B-Instruct, one PPO recipe, and one retrieval setup. Although the paper evaluates five QA datasets, this does not establish robustness across model families, scales, training seeds, or optimization settings.

Questions for authors
  1. Which conclusions are new in addition to OASES’s existing co-training and process-reward ablations?
  2. Do the main comparisons hold across training seeds, base models, and matched computational budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.