acceptodds
Under review as a conference paper at ICLR 2027

Bring the Answer into View: Answer-Bearing Frontier Training for Search Agents

Abstract

Outcome reinforcement learning directly optimizes final-answer correctness, but its terminal signal conflates a trajectory that never acquires the needed information with one that retrieves it but fails to use it. Fine-grained rewards offer denser credit, yet a search trajectory mixes exploration, acquisition, verification, and stopping, for which one proxy need not have a consistent meaning. We study a complementary principle: apply a simple, clear reward at a position where its intended consequence is directly observable. The Answer-Bearing Frontier (ABF) is the state immediately before a tool observation first contains the reference answer; the realized transition witnesses that answer-bearing information is reachable in one step. Answer-Bearing Frontier Training restores these states, samples alternative one-decision continuations, and rewards immediate acquisition, while full-trajectory Outcome updates optimize reaching useful states, integrating information, stopping, and answering. With local supervision only at selected turns, ABF Training achieves 37.4% and 48.8% average exact match across six benchmarks at 2B and 9B, with margins of +2.3 and +1.7 points over the strongest comparators and leads 9 of 12 dataset–scale settings. Relative to Search-R1 at 2B, answer-bearing coverage rises from 53.8% to 61.8%, while exact match given such information changes from 51.9% to 53.0%. The gains extend beyond the answer types used for local training. These results show that a strong training position can turn a narrow local reward into stable end-to-end gains.

Primary area:
reinforcement learning
Keywords:
Search Agents, Reinforcement Learning, Temporal Credit Assignment, Process Supervision, Information Acquisition

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Cg2WOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Answer-Bearing Frontier (ABF) Training for multi-turn search QA. It mines states immediately before the first answer-bearing tool observation, restores their histories, samples one-decision continuations, and rewards immediate acquisition with search, document, and terminal rewards . ABF updates alternate with full-trajectory Outcome GRPO updates. Across 6 benchmarks at 2B and 9B, ABF reports the highest average EM, 37.4% and 48.8% respectively.

Critical strengths
  1. The author gives clear problem framing and precise ABF definition, the first answer-bearing transition is formalized with a witness action and frontier state.
  2. Coherent local reward design for search, document opening, and terminal answers, including discouraging premature answering.
  3. Alternating ABF and Outcome updates are well motivated, with rollout expansion/compression dynamics shown across training cycles.
Critical weaknesses
  1. The acquisition predicate is lexical and restricted to named-entity training answers. The paper acknowledges misleading matches for numerical/Boolean answers and misses paraphrases, so frontier selection and rewards may not track semantic acquisition.
  2. Cost accounting is easy to misread. The 12.1% sampling and 19.3% work ratios use selected-prefix lower bounds; full recorded mining is 88.4% and 69.2%, with retrieval, validation, and overhead excluded.
  3. The authors only conduct experiments on the Qwen3.5 series and do not validate their method on the weaker Qwen2.5 base model. Since most baseline [1, 2] comparisons in the paper are evaluated on Qwen2.5-7B-Instruct, the generalization ability of the proposed approach cannot be fully verified.

[1] Wang et al. Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents. ICLR 2026.

[2] Ma et al. TSPO: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization. ACL Findings 2026.

Questions for authors
  1. How does ABF behave if the lexical answer-bearing predicate is replaced by a semantic verifier or LLM judge, and what happens on numerical/Boolean questions excluded from training?
  2. What are matched full-compute comparisons against baselines, given that the reported ABF cost ratios are lower bounds or selected-prefix proxies?
  3. Could the authors supplement experiments based on the Qwen2.5 base model and conduct comparative evaluations under the experimental settings strictly aligned with baseline methods such as IGPO?
6 comments
AuthorsOpenReview ↗

Response to reviewer Cg2W

Thank you for your feedback. We took your concerns seriously and completed substantial additional experiments. We hope the new evidence and manuscript revisions address your concerns.

Q1: Can a semantic judge replace lexical matching, and how does ABF perform on numerical/Boolean questions?

  1. What happens when a semantic judge replaces lexical matching? It brings a small improvement, with extra cost. We added a training experiment using a frozen Qwen3.5-2B judge for frontier selection and local rewards. After two rounds, it matches the lexical run's EM (33.33%) and improves F1 from 37.58 to 39.94, but needs two additional GPUs and longer training. We added these results to Section 5.1 and Appendix E.

We also added independent semantic evaluation with a Qwen3-32B judge on all 2,130 evaluation questions, including Boolean questions. ABF remains higher on all four acquisition metrics, now included in Table 2:

Semantic metricSearch-R1ABF
First-search hit rate (%)18.723.7
All-search hit rate (%)14.517.1
Supporting documents / rollout0.550.87
Supporting open rate (%)16.722.0

Together, these results show that lexical matching provides a useful training signal despite its noise, while semantic supervision offers a trade-off between quality and cost.

  1. Does ABF help on numerical and Boolean answers excluded from local training? Yes. The lexically trained ABF improves F1 from 79.6 to 84.7 on Yes/No questions and 27.8 to 29.7 on Number/Quantity questions (Table 3). This supports transfer beyond its training answer types. The judge-training pilot used a separate validation set without Boolean or purely numeric reference answers.

Q2: Is ABF's cost advantage preserved when frontier mining is included?

Yes, in sampling tokens, even including all recorded mining. A local update uses 1.509M tokens, versus 1.706M for a paired Outcome update. Training one decision saves enough sampling to offset the mining overhead.

The training curves also show better accuracy at similar sampling budgets: 47.6% validation EM at 312M tokens for ABF, versus 41.9% at 308M for Search-R1 (Figure 5(a)).

We revised the cost description to make the accounting explicit. Figure 5 counts selected frontier prefixes, fresh local continuations, and Outcome rollouts, including model-call inputs and outputs. The 12.1% ratio uses this accounting; 88.4% includes all recorded mining. Appendix E gives both calculations. These are sampling costs, excluding optimizer updates, retrieval, and evaluation.

Q3: Do the gains hold on Qwen2.5 and in the baselines' original settings?

  1. Does ABF improve the weaker Qwen2.5 model? Yes. We added a Qwen2.5-7B-Instruct experiment, comparing ABF and Search-R1 from the same Base with the same checkpoint-selection rule:
DatasetSearch-R1 EMABF EMSearch-R1 F1ABF F1
HotpotQA26.2727.2337.1838.78
2Wiki22.8023.6027.4131.36
MuSiQue5.409.8013.1918.24
InfoSeek41.0042.3351.4253.44
Bamboogle15.2031.2023.4037.50
PopQA48.0046.2055.3455.55
Average26.4430.0634.6639.15

ABF gains 3.62 mean EM and 4.49 mean F1 points, with higher EM on five of six datasets, supporting its benefit beyond Qwen3.5.

  1. What happens in a baseline's original setup? We added an experiment using DeSA's official code and native Qwen2.5-3B setup, retaining its retriever, tool interface, and full-parameter training. With 160 local + 160 Outcome updates, ABF reaches 37.63 mean EM, versus 35.74 for released DeSA Stage 2 across seven datasets. We estimate this uses roughly 20% of the cost of DeSA's 1,000-step training budget.

IGPO's live Google-search environment is harder to reproduce, so we have not run an original-setting comparison. In our shared setup, we retain its information-gain supervision; Appendix C.1 explains the adaptations.

W1–W3: Addressed in Q1 (lexical reliability and answer types), Q2 (cost), and Q3 (Qwen2.5 and original-setting comparisons).

I appreciate the authors' rebuttal, which provides helpful additions regarding semantic judges, Qwen2.5 baseline experiments, and cost metric clarifications. However, several core concerns remain unresolved: the numerical/Boolean evaluations are not explicitly validated within semantic frontiers or rewards, and the DeSA setting is not strictly aligned. Additionally, the paper's presentation and figure still require refinement. Therefore, I maintain my original score.

Thanks for getting back to us. We'd like to clarify three points so we can better understand and address your remaining concerns.

1) Numerical and Boolean questions. Do you mean evaluating evidence acquisition on these questions with a semantic judge, or including them in local training with semantic frontiers and rewards?

For semantic evaluation, we have just broken down the existing Qwen3-32B judgments by answer type. Below are the results for Search-R1 and our original, lexically trained ABF, using the metrics in Table 2:

Answer typeSemantic metricSearch-R1ABF
BooleanFirst-search hit (%)88.693.8
BooleanAll-search hit (%)79.878.9
BooleanSupporting docs / rollout3.294.25
BooleanOpen hit (%)61.472.1
NumericalFirst-search hit (%)4.55.4
NumericalAll-search hit (%)7.87.0
NumericalSupporting docs / rollout0.340.44
NumericalOpen hit (%)9.212.9

ABF has higher first-search hit rates, more supporting documents, and higher open hit rates in both groups, though all-search hit rates are slightly lower.

If you mean including these question types in local training with semantic supervision, that is beyond this paper's claims and remains a direction for future work.

2) DeSA settings. We aligned the added experiment with DeSA's original setup, using its official code, Qwen2.5-3B model, NQ+HotpotQA training data, Wiki-18/E5 retrieval with top-3 results, tool interface, and full-parameter optimization. ABF's local training and alternating Outcome schedule are implemented within this setup. With the same evaluation questions and official scoring, ABF reaches 37.63 mean EM vs. 35.74 for DeSA's released Stage 2 model across seven datasets. Could you tell us which specific setting you still consider misaligned, or what concern remains about this comparison?

3) Presentation and figures. Could you point us to the passages or figures you found unclear in the revised version, and tell us what was hard to follow? We're happy to revise them, but need specific examples to understand what needs changing.

Before undertaking further experiments, we would appreciate knowing which remaining concern they should address and what result would resolve it.

Reviewer_pqyHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets a credit-assignment gap in RL for multi-turn search agents. Outcome-only training (Search-R1) cannot distinguish a trajectory that never retrieves the answer from one that retrieves it but fails to use it, and dense process rewards apply a single proxy to steps whose objectives differ (bridge retrieval, acquisition, verification, stopping), which the authors argue introduces noise. The proposal, Answer-Bearing Frontier (ABF) Training, selects one specific state per trajectory, the state immediately before the first tool observation that lexically contains the reference answer, restores its history, samples eight one-step continuations, and rewards each by whether the resulting observation exposes the answer string.

Critical strengths
  • The paper makes a clear and testable claim: where a local reward is applied matters independently of what the reward is. Rather than relying on end-to-end numbers alone, the authors support this with a controlled ablation that fixes the reward and schedule and changes only the supervision position.
Critical weaknesses
  • The ablations isolate where supervision is applied but not how: the acquisition reward, the strict one-decision truncation, and the alternating schedule are motivated but untested. The most informative control would branch from the same frontiers and score full continuations with the outcome reward (BranPO/PivotRL-style). We recognize this is roughly 7–8× the per-step cost of an ABF update, but at the scale of the existing ablations (2B, validation, ~100 Outcome steps, possibly 4 continuations) it is comparable to a fraction of one Search-R1 run. Either result would be informative: parity would recast the contribution as a cheaper route to branching gains, while a gap would support the local reward itself. If outcome-scored branches suffer advantage collapse at frontiers, reporting that would also be useful. A cheaper partial check is a two- or three-decision continuation with the same reward, which would at least show whether strict truncation matters.

  • A frontier exists only if the current policy exposes the answer somewhere in the trajectory (61.1% of mined trajectories). The questions where acquisition is weakest never receive ABF supervision

Questions for authors
  • Aligning all methods to one setup is understandable for a fair comparison, but the resulting baseline numbers are quite far from what their original papers report. A gap this large makes it hard to tell whether the "previous intermediate supervision does not uniformly help" finding reflects the methods or the setup. Could the authors comment on whether their implementations reproduce the original gains in the original settings, or on which setup choices they believe are responsible?
  • For the random-position ablation (Fig. 2a), was the same depth-stratified sampling (2:2:3:1) applied to the randomly chosen positions, and were all buckets fillable?
  • Fig. 3(b) shows ABF loses bearing on a set of questions, costing a net 23 correct answers, offset by a 25 gain in Yes→Yes. The paper highlights the No→Yes cell but does not analyze why acquisition regresses on others
  • The current setup is short-answer QA over a fixed Wikipedia chunk corpus with two tools, where the reference answer appears verbatim in retrieved text. How do the authors envision extending ABF to settings where this assumption breaks? In particular: (i) for open-web or long-page environments (e.g., BrowseComp-style tasks), where the lexical predicate may fire spuriously on long pages or only after several steps of verification, would the frontier definition need a semantic or clue-based predicate, and how would that affect cost? (ii) For derived answers (counts, comparisons, dates computed from evidence) no single observation contains the answer, so no frontier exists; could the frontier be redefined around intermediate evidence (e.g., the last required bridge entity), and would the one-decision reward still be well-defined there?
3 comments
AuthorsOpenReview ↗

Response to reviewer pqyH

Thank you for your feedback.

Q1: Which setup differences might explain why the baselines' original gains do not carry over to our shared setting?

We believe GRPO rollouts per question, search results per query, and the terminal reward are the main factors. These settings interact closely with the reward design of DeSA, TSPO, and IGPO.

SettingSearch-R1 (GRPO)DeSATSPOIGPOOur shared setup
Rollouts / question (G)5551616
Search results / query3351010
Terminal rewardEMEMEMF1EM
  • DeSA: Its binary recall reward checks whether the answer appears anywhere in retrieved text. Returning 10 rather than 3 results may make positive rewards more common and weaken reward contrast. Its Fig. 6(a) likewise shows recall rising while final-answer EM falls.
  • TSPO: It rewards retrieval in all-wrong groups, frequent at G=5 (its Fig. 2). Our G=16 may produce more mixed-reward groups, reducing the need for this correction.
  • IGPO: We retain information-gain supervision but replace terminal F1 with EM for a common final-answer objective. Unlike F1, EM gives no partial credit, potentially making terminal rewards sparser and affecting training.

To further validate ABF's effectiveness, we added an experiment using the official code and original Qwen2.5-3B setup of DeSA, the closest method to ours. Across seven datasets, ABF achieves 37.63 mean EM, compared with 35.74 for DeSA's released model, after 160 local + 160 Outcome updates at an estimated 20% of DeSA's 1,000-step training cost (Cg2W, Q3).

Q2: Did Random bearing use the same depth allocation, with all buckets filled?

Yes. The original Random bearing comparison used Frontier's 2:2:3:1 allocation, and all buckets were filled.

Q3: Why does acquisition regress on some questions, costing 23 net correct answers?

ABF encourages search but can exhaust the budget: among 80 Yes→No questions, 13 ABF trajectories hit the tool limit versus 4 for Search-R1. Sampling randomness may also contribute. Despite these losses, ABF still gains 85 net correct answers overall, making this trade-off acceptable (Figure 3(b)).

Q4: How could ABF extend to broader settings?

  1. Would the frontier need a semantic or clue-based predicate, and how would this affect cost? Yes, when lexical matching is unreliable, a semantic or clue-based predicate may be needed. We added a training experiment replacing lexical matching with a semantic judge: F1 rose from 37.58 to 39.94, with unchanged EM, but required two extra GPUs and longer training. This shows a modest benefit at extra cost (Section 5.1; Appendix E).

  2. Could the frontier be redefined around intermediate evidence (e.g., the last required bridge entity), and would the one-decision reward still be well-defined there? Our added experiment shows that a semantic judge can replace lexical matching in training, suggesting that intermediate evidence could serve as a target too. Our current idea, not yet tested, is to have steps declare their purpose—[Synthesis], [Retrieval], or [Verification]—and use purpose-specific rewards for GRPO. Alternatively, a judge could identify acquisition of a key intermediate fact, defining the frontier and rewarding new actions that acquire it. Whether this works depends on the reward signal-to-noise ratio and whether the gains justify the cost; both need empirical study.

Following your suggestions, we added experiments to address these weaknesses.

W1: The one-decision training design needs a direct comparison.

One-step acquisition beats full-suffix Outcome training by 8.1–14.3 EM points. We ran your suggested comparison from identical saved frontier states, with four fresh continuations per state and the same outer Outcome schedule. The arms use one-step acquisition reward and full-suffix terminal reward, respectively. Validation EM (%):

Round1234
One-step acquisition26.1938.5737.6238.10
Full-suffix Outcome16.6724.2929.0530.00

This supports direct training of the next acquisition decision (Section 4.3, Figure 2(c)).

W2: Questions without a frontier miss local supervision.

Initially ineligible questions can enter local training after pool refreshes; all questions remain available to Outcome updates. The reported 61.1% is a trajectory-level rate. We tracked initially ineligible questions through ten refreshes:

Initially ineligible questionsCount
No legal frontier in first pool473
Sampled again later217
Later obtained a legal frontier72
Subsequently used in ABF updates30

Of the 217 sampled again, 72 (33.2%) became eligible and 30 received local training. The other 256 were not sampled again. Thus, eligibility can change (Appendix A).

Thank you for the detailed rebuttal and the additional experiments. My main concerns have been addressed. I think adding evaluation-time tool calls per question and the limit-hit rate to Table 2 would help readers see the search-cost trade-off.

I am glad to hear that your concerns have been addressed. Thank you for your positive assessment and helpful suggestion. We have revised the manuscript to include the following statistics in Table 2: ABF uses 5.37 tool calls per question versus 4.60 for Search-R1, with limit-hit rates of 15.01% and 11.47%, respectively.

Reviewer_N2fDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Answer-Bearing Frontier (ABF) Training, a method for improving search agents through targeted intermediate reinforcement learning. The authors identify the state immediately preceding the first tool observation containing the ground-truth answer, restore the agent to that state, and sample alternative one-step actions. These actions receive rewards based on whether the subsequent tool observation contains the answer. Local ABF updates are alternated with full-trajectory outcome-based reinforcement learning.

The method improves average exact-match accuracy over the strongest reported baselines by 2.3 and 1.7 percentage points at 2B and 9B, respectively, across six QA benchmarks. The authors also report higher answer-bearing retrieval rates and ablations supporting the importance of frontier selection and depth stratification.

While the method is reasonably motivated and the empirical improvements are encouraging, I have reservations about the novelty of the proposed supervision signal, whether the experiments adequately isolate the contribution of frontier selection, and the generality of the approach.

Overall Assessment

This paper presents a plausible and reasonably well-motivated technique for improving search-agent training. The reported accuracy improvements and the controlled training analyses suggest that targeted local supervision can be useful.

However, I am not convinced that the paper sufficiently establishes the novelty or incremental value of its specific method relative to recent work on intermediate search / proccess rewards.

The acquisition signal is a familiar lexical heuristic with known limitations, and the main state-selection ablation does not adequately distinguish the value of the proposed frontier from the simpler principle of applying acquisition rewards before the answer has already been retrieved. The reliance on the same lexical criterion for training and acquisition evaluation further limits the strength of the mechanistic evidence.

Although the empirical gains are encouraging, the lack of fully controlled comparisons against plausible state-selection alternatives, together with limited statistical characterization of the strongest baselines and the absence of compute-matched evaluation, makes it difficult to attribute the improvements confidently to the paper's claimed methodological contribution.

I therefore lean Weak Reject. A stronger set of pre-acquisition state-selection ablations and independently validated evidence-acquisition measurements would materially improve my assessment.

Critical strengths

Clear motivation and straightforward implementation. The paper identifies a legitimate difficulty with applying intermediate rewards uniformly across search trajectories. The idea of selecting states where a specific local objective is immediately achievable is intuitive and computationally convenient.

Positive empirical results. ABF improves average exact match across six benchmarks at both evaluated model scales. The gains over outcome-only training are meaningful, particularly on InfoSeek at 2B. The evaluation also includes several relevant process-supervision baselines.

Useful ablations and behavioral analyses. The authors examine first-occurrence selection, depth-stratified frontier sampling, and the interaction between local and outcome updates. The increase in answer-bearing coverage is consistent with the proposed acquisition objective.

Implementation transparency. The paper provides relatively detailed training specifications and discusses the costs of frontier mining, including the limitations of selected-prefix sampling estimates.

Critical weaknesses
  1. Limited methodological novelty relative to existing process-supervision approaches.

The central local reward is based on detecting occurrences of the ground-truth answer in retrieved observations. This is a relatively simple heuristic, and related forms of answer-recall or answer-occurrence supervision have already been explored in search-agent reinforcement learning.

In particular, DeSA uses answer-recall-based rewards, while TSPO explicitly exploits the first retrieved occurrence of the ground-truth answer to assign intermediate credit. Other work, including LeReT, IGPO, and Fathom-DeepResearch, has explored alternative methods of providing more informative intermediate supervision for search and retrieval.

The distinguishing contribution of this paper is therefore not the answer-matching reward or the first-occurrence criterion individually, but their combination with frontier-state restoration, one-decision resampling, and alternating outcome optimization.

This is a legitimate methodological distinction. However, given the substantial existing literature on process-level supervision and intermediate-state optimization, I do not think the current experiments adequately establish that this particular combination represents a significant advance over simpler alternatives.

  1. The principal state-selection ablation does not isolate the claimed benefit of the answer-bearing frontier.

The paper argues that the state immediately preceding first answer acquisition is particularly suitable for local training because the answer is not yet available, while the recorded trajectory demonstrates that acquiring it in the next action is possible.

This is a reasonable motivation. However, the main ablation in Figure 2(a) compares the proposed frontier against a randomly selected answer-bearing position from the same eligible trajectory.

The comparison is not sufficiently controlled for the paper's central claim. Random answer-bearing positions may occur after the correct answer has already appeared in the agent's observations. At such positions, additional answer retrieval may be unnecessary, and useful actions may instead involve verification, reasoning, or final-answer generation.

Consequently, the substantial performance difference establishes that rewarding answer acquisition after the answer has already been observed may be counterproductive. It does not demonstrate that the state immediately preceding first answer occurrence is superior to other reasonable pre-acquisition states.

A more convincing ablation would compare the proposed frontier against randomly selected pre-acquisition states, earlier states from the same trajectory, or other state-selection criteria, while keeping the local reward, number of training examples, and optimization budget fixed.

This is particularly important because the choice of training position is presented as one of the paper's principal contributions.

  1. Lexical answer occurrence is an imperfect proxy for meaningful evidence acquisition.

The proposed reward relies on lexical matching of the canonical answer or a filtered alias in retrieved observations. For search actions, the reward increases with the number of retrieved documents containing the answer string.

However, the occurrence of the correct answer in a document does not establish that the document provides relevant or sufficient evidence for the question. An answer may appear incidentally, in an unrelated context, or alongside contradictory information.

The reward may therefore encourage retrieval of documents containing the answer string rather than documents that establish the relevant factual relationship.

The authors acknowledge this limitation and restrict local training to named-entity answers. Nevertheless, this remains an important concern because the same matching criterion is used both to construct the training signal and to measure answer-bearing coverage in the behavioral analysis.

As a result, improvements in answer-bearing hit rates partly establish that the model has learned to optimize the proposed lexical proxy. They do not independently establish improved acquisition of semantically relevant or reliable evidence.

An evaluation using independently verified supporting passages, or an analysis of the frequency and consequences of false-positive answer-bearing matches, would strengthen the mechanistic claims.

  1. Limited applicability of the proposed local supervision to broader search and reasoning tasks.

The local training procedure is restricted to questions with named-entity answers because lexical matching becomes less reliable for other answer types.

While the authors report improved performance on non-entity questions at evaluation, these results demonstrate transfer of the learned policy rather than direct applicability of the proposed reward to those question types.

It remains unclear how this approach would extend to open-ended questions requiring explanations, evidence synthesis, numerical reasoning, or reconciliation of conflicting sources, where successful information acquisition cannot be reliably identified through the occurrence of a single reference-answer string.

This limitation is particularly relevant for general-purpose search agents, where obtaining the answer often requires combining multiple pieces of evidence rather than retrieving a passage containing the final answer.

I do not consider the named-entity restriction inherently disqualifying, but I believe the paper should more clearly delimit its contribution as an acquisition-training method for settings with readily matchable reference answers.

  1. Insufficiently controlled evidence of superiority over competing training methods.

ABF and Search-R1 are evaluated over three training seeds, whereas DeSA, TSPO, and IGPO are evaluated using only one run per model scale.

Given that the improvements over the strongest competing baselines are 2.3 and 1.7 exact-match points, the absence of repeated runs for those methods makes it difficult to assess the reliability of the reported margins.

Moreover, the comparison uses a common optimizer-step budget rather than matched total training compute. ABF updates require frontier mining, state restoration, and additional local continuations, which incur costs different from full-trajectory updates.

The authors provide a useful discussion of these costs in Appendix E and explicitly acknowledge that their analysis is not a complete compute-matched comparison. Nevertheless, the practical advantage of the method over stronger process-supervision baselines remains uncertain without such an evaluation.

Finally, ABF combines several design choices, including local reward shaping, frontier selection, depth stratification, and alternating local and outcome updates. The current ablations do not fully disentangle these components. In particular, the effectiveness of the alternating schedule relative to sequential ABF and outcome training is not convincingly established.

Questions for authors

Can the authors compare ABF with alternative pre-acquisition state-selection strategies, rather than randomly sampled positions at which the answer may already have been retrieved? This seems necessary to isolate the central frontier-selection claim.

How frequently does the lexical answer-bearing criterion identify documents that contain the answer string without actually supporting the answer to the question? Can the acquisition improvements be validated using an independent relevance or evidence-sufficiency measure?

How much of the gain can be attributed specifically to frontier restoration and one-decision resampling, as opposed to applying a simple answer-occurrence reward at suitable intermediate states?

Do the improvements over the strongest process-supervision baselines remain consistent across multiple training seeds and under matched total training compute, including frontier-mining overhead?

Can the authors clarify the intended scope of the method beyond named-entity QA? In particular, how would the frontier and its reward be defined when the answer requires synthesizing evidence from multiple sources rather than retrieving an explicit answer string?

1 comment
AuthorsOpenReview ↗

Response to Reviewer N2fD

Thank you for your feedback and willingness to reconsider. We hope the added experiments and manuscript revisions address your concerns.

Q1: Does frontier selection outperform alternative selectors?

Yes. Frontier performs best throughout our added comparisons. Random pre-frontier samples states strictly before the frontier; Random later bearing samples Random bearing candidates with the frontier excluded. Both retain the local reward and training schedule. Validation EM (%) by Outcome updates after Base:

Selector20406080100
Frontier (ABF)28.5733.3339.0542.8642.86
Random pre-frontier20.0030.9529.5234.7632.38
Random later bearing20.9530.0026.6732.3831.43
Random bearing24.7025.8027.2031.0029.80

This supports training at an unmet acquisition opportunity (Section 4.3, Figure 2(a)).

Q2: How noisy is lexical matching, and do gains survive semantic evaluation?

Lexical matching is noisy, but ABF's acquisition gains persist under independent semantic evaluation. Our added audit found that 35.88% of Search-R1's and 39.87% of ABF's lexical matches did not support the answer. ABF remains ahead under both criteria (Section 5.1, Table 2). Values below are lexical → semantic:

MetricSearch-R1ABF
First-search hit (%)25.1 → 18.731.3 → 23.7
All-search hit (%)21.0 → 14.525.8 → 17.1
Documents / rollout1.03 → 0.551.69 → 0.87
Open hit (%)24.8 → 16.732.1 → 22.0

We also added a separate training experiment using a semantic judge instead of lexical matching. After two rounds, EM remained unchanged (33.33%), while F1 improved from 37.58 to 39.94, with two additional GPUs and longer training. The judge offers a modest improvement at extra cost; lexical matching remains a useful, cheaper training signal (Section 5.1; Appendix E).

Q3: What does one-decision training add beyond training at intermediate states?

One-step acquisition beats full-suffix Outcome training by 8.1–14.3 EM points. Our added comparison uses identical saved frontier states, four fresh continuations per state, and the same outer Outcome schedule. Validation EM (%):

Round1234
One-step acquisition26.1938.5737.6238.10
Full-suffix Outcome16.6724.2929.0530.00

With restoration shared, this supports local training alongside Q1's selection evidence (Section 4.3, Figure 2(c)).

Q4: Do gains persist across runs, and does mining erase the cost advantage?

ABF retains its lead across runs, and local sampling savings remain including mining. We updated DeSA, TSPO, and IGPO with multi-run averages. Six-dataset mean EM (Table 1):

MethodQwen3.5-2BQwen3.5-9B
Search-R133.646.9
DeSA35.446.0
TSPO34.147.4
IGPO32.044.8
ABF37.448.8

ABF leads the strongest baselines by 2.0/1.4 points and ranks first in 9/12 dataset–scale settings.

Including all recorded mining, a local update uses 1.509M sampling tokens vs. 1.706M for a paired Outcome update (Appendix E). Figure 5(a) also shows 47.6% EM at 312M tokens vs. 41.9% at 308M under selected-prefix accounting. These comparisons measure sampling efficiency.

Q5: How could ABF handle multi-source synthesis?

The training procedure can use semantic criteria; synthesis remains a possible extension. ABF already gains 3.4 F1 points on non-entity questions (Section 5.3), and Q2 tests semantic supervision.

The agent could declare each step's purpose: [Synthesis] (what to combine and why), [Retrieval], or [Verification]. An LLM would identify steps that fulfill their purpose and help solve the task. We would revisit those states, sample alternative next actions, and use GRPO to favor actions that fulfill the goal. This proposal is untested.

We added experiments addressing the weaknesses you identified:

W1: Insufficient evidence of an advance over simpler alternatives.

As you suggested, we added pre-acquisition controls (Q1) and independent semantic evaluation (Q2). These support frontier selection and semantic gains; Q3 supports one-step training.

DeSA trains full-trajectory recall; ABF targets the next acquisition decision. In DeSA's native Qwen2.5-3B setting, ABF reaches 37.63 mean EM vs. 35.74 for released DeSA Stage 2 across seven datasets after 160 local + 160 Outcome updates, at an estimated 20% of DeSA's 1,000-step cost (Cg2W, Q3). This supports targeted local training for higher accuracy at lower cost.

W2–W5: Added controls support frontier selection (Q1); semantic evaluation confirms acquisition gains (Q2); non-entity results support transfer (Q5); repeated runs and mining-inclusive sampling costs strengthen the comparison (Q4).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Additional Experiments

Summary of Revisions and Additional Experiments

We thank all reviewers for their constructive feedback. In response, we have revised the manuscript and added experiments addressing the main concerns:

  • New controls support frontier selection and one-step training. Frontier outperforms the added pre-frontier and later-bearing selectors. On shared frontier states, one-step acquisition gains 8.1–14.3 EM points over full-suffix Outcome training (N2fD, Q1/Q3).
  • Acquisition gains persist under semantic evaluation. We added independent LLM-judge evaluation and a semantic-reward training pilot, supporting the acquisition gains and examining the quality–cost trade-off (Cg2W, Q1).
  • Additional runs and Qwen2.5 experiments support robustness. We updated three baselines with multiple-run averages and added Qwen2.5-7B and DeSA-native comparisons. In DeSA's original setup, ABF achieves 37.63 vs. 35.74 mean EM (N2fD, Q4; Cg2W, Q3).
  • Sampling savings remain when mining is included. With all recorded mining, a local update uses 1.509M tokens vs. 1.706M for a paired Outcome update. We clarified this accounting in the revision (Cg2W, Q2).

Our individual responses provide the detailed results. We hope these additions address the concerns and welcome further feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.