acceptodds
Under review as a conference paper at ICLR 2027

One Answer to Grade Them All? Reference-Based Evaluation and Its Optimization Effects in Search Agents

Abstract

Search agents have become increasingly capable of tackling complex questions, with reliable judgments of answer correctness providing the basis for assessing their capabilities and guiding further learning. Search-agent evaluation and training commonly rely on reference answers, although recent work incorporates verified alternative answers into training rewards. However, the implications of incomplete reference coverage for evaluation and supervision remain insufficiently understood in more challenging web-search tasks. Questions combining multiple constraints may still admit valid solutions beyond the references, causing erroneous rejection and potentially misleading supervision. To examine this problem, this study uses evidence-based verification to quantify grading discrepancies and analyzes their sources through question constraints and supporting evidence. Policy updates under original and corrected rewards are then compared to examine how grading errors affect supervision of answers and search actions. Extensive experiments across five search benchmarks show that reference-based evaluation can underestimate performance by rejecting valid answers, with verified revisions increasing the acceptance rate by 2.12 pp. Source-based attribution identifies reference omissions as a distinct error category arising from one-to-many relations or unspecified selection rules. When used as rewards, these misjudgments can lower the likelihood of valid answers and useful search actions, depending on response content, group composition, and search context. These findings motivate checking for competing valid solutions when constructing questions and references for evaluation and learning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
search agent, reference-based evaluation, reward bias

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_BRNcOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines false rejections of valid answers under reference-based evaluation and their local optimization effects when these judgments are used as rewards. Evidence-based verification across five search benchmarks yields a 2.12-percentage-point increase in acceptance rate, while source-based attribution distinguishes strict reference omissions from other grading errors. The authors then hold the initial model and trajectory groups fixed, correct a target reward, and measure how a single parameter update changes the likelihood of valid answers and useful search queries. Additional comparisons examine response content, group composition, and evaluation context.

Critical strengths
  1. The distinctions between grading errors are practically useful. The paper carefully separates reference omissions, defective references, equivalent expressions, and interpretation differences, while assessing task-answer and full-response validity separately. This helps identify what requires correction rather than attributing every disagreement to incomplete answer coverage.

  2. The paired-update design provides a reasonable basis for local analysis. Holding the model and trajectories fixed while changing only the target reward helps isolate the intervention’s total effect. The authors distinguish relative improvement from recovery above the initial likelihood and report inconsistent correction effects rather than interpreting local measurements as evidence of long-term training gains.

Critical weaknesses

1. The substantive connection between reference omissions and the gradient analysis is insufficiently established. The optimization analysis primarily concerns binary reward corrections on fixed trajectories, and its central derivations apply equally to false rejections arising from other causes. Although the authors report some results by error category, it remains unclear whether reference omissions exhibit systematic structure in the affected answers, search paths, or gradient effects beyond general reward errors. Both parts have value individually, but the additional insight obtained by connecting them is not yet sufficiently clear.

2. The likelihood changes are observed, but the gradients’ specific contributions and functional effects remain unclear. The paper defines alignment between trajectory gradients and the evaluated target, yet mainly reports the resulting likelihood changes. This makes it difficult to determine whether an effect comes from the target trajectory, companion reweighting, or interference between different content segments. Moreover, changes in a particular query string’s likelihood do not establish changes in search-intent preference, and the existing next-action check finds no consistent selection benefit. The optimization section therefore remains closer to a demonstration of local phenomena than a sufficiently validated mechanism analysis.

Questions for authors

1. Error specificity. What additional optimization insight does reference omission provide beyond general false-rejection effects? A comparison with other error types or randomly introduced false rejections, under matched group sizes and reward counts, would help clarify this.

2. Gradient contribution. Could you identify which gradient components drive the observed suppression or improvement—for example, the target versus companion trajectories, or search versus final-answer tokens? Comparing first-order predictions with actual likelihood changes would also help validate the mechanism.

3. Functional effects. Beyond changes in particular strings’ likelihoods, do the updates affect preferences for valid answers or useful search intents? Candidate-ranking tests or an extension of the next-action check in Appendix F.3 could help establish this.

1 comment

I downgrade my scores due to the lack of response from the authors.

Reviewer_2uyWOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors focus on the reference-based nature of evaluation in search-augmented question answering settings. They demonstrate that the poor coverage of references can underestimate model performance on various commonly used datasets such as BrowseComp-zh and make an argument for how this underestimation can also impact training.

Critical strengths
  • I think the limitations of reference-based evaluation in both training and evaluation for search-augmented question answering is a very valuable thing to study, as I have also found this to be a limitation in my own prior work on search models (though this wasn't the focus of such work). Both in terms of how this can affect training and our understanding of model performance, but also in light of reward hacking incidents and how unsolvable questions (or noise) could cause unintended consequences.
  • The result on Section 3 is compelling in terms of demonstrating noise on existing datasets.
Critical weaknesses

While I do find the main motivation to be exciting, I think the execution of the paper is sadly wanting. There are two separate sets of flaws for Section 3 and Sections 4-5.

In the case of Section 3, the main issue is of clarity and lack of depth:

  • The paper does not reveal most of the important experimental details on the main text. What are the models that are evaluated? This should not be relegated to figures/tables and should be specified explicitly. Likewise, when rejected answers are analyzed, what are these "three independent verifiers"? How do they search for external evidence? The experimental setup for the main result demands more detail.
  • There are various turns of phrase that I found difficult to parse throughout the paper. Some examples are "Accepted gives unanimous acceptances as a percentage of eligible rejections." Telegraphic phrases like "supported task answers in 71% of verifier-supported rejections" in Section 3.3 are likewise confusing.
  • While the fact that benchmarks underestimate performance is compelling and intuitive, the paper does not investigate the consequences further. Do model rankings change? Do we learn anything new about model behavior and limitations that we did not before from this improved evaluation?

Section 4 and 5, however, are where the paper's critical flaws lie. Section 4, firstly, is trivial and arguably does not need to be a section. A change in reward will cause a change in GRPO advantages. This follows from the fact that GRPO advantages are computed from group rewards and this will naturally have consequences on model answer likelihoods, etc. This does not need two pages dedicated to it. The question of how training would then be affected by this kind of performance underestimation is interesting, but Section 5 does not give a meaningful answer to this. If the authors want to assess impacts on training, the authors should train models over multiple steps (say, 200-300) in an actual reinforcement learning experiment. An analysis of an individual gradient step is not meaningful, especially when only 3% of groups have a change in advantage (as noted in Section 4). This is a small amount of noise and it is not clear to me (though I would like to know) whether this small noise will actually have meaningful impact on performance.

These limitations of Sections 4 and 5, which are fundamental to the paper, is why I am giving the scores I am giving.

Questions for authors

A proper reinforcement learning would go a long way to answering some of the questions the paper asks. One relatively cheap option would be to corrupt the ground-truth answers such that the ground-truth answer is a completely unrelated fact that a model is unlikely to find. This would induce an underestimation of performance. Alternatively, the authors can attempt to devise a lightweight way to perform evidence-based answer validation, which would then require a large enough training set featuring multiple valid responses.

1 comment

I am maintaining my scores due to the lack of response from the authors.

Reviewer_JEwVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper first claims that many complex questions may admit many answers, while not all of them are in the reference solution. This particularly affect training search agents, as the reward is calculated from such incomplete reference, potentially rejecting vaild answers. Through experiments on five benchmarks, they show that such standard rewards lower the agent's performance while corrected reward improve the acceptance rate by 2.12%. The corrected reward is implemented by evidence-based verification of the candidate answers.

Critical strengths
  • The challenge of incomplete reference answers is well-aware in various QA-formatted tasks.
  • The author conducts various experiments to provide evidence for their argument,
Critical weaknesses
  • The specific challenge in web-search tasks is not clearly stated, and it is unclear why A2Search[1] is insufficient for the proposed challenges. The methodological novelty beyond A2Search is also unclear, given that the verification pipeline in this paper also relies on evidence-based checking of candidate answers.
  • The extra cost (time latency, extra tokens spent) of the proposed verification pipeline is not reported. This is especially critical for RL training.
  • It is unclear how many data points are used in Section 5. Moreover, the effect of the reward correction on downstream tasks (e.g., HotpotQA) is not evaluated.
  • It seems that among all the appendices (A–H), only D.2, E.1, G.1 and H are referenced in the main text (Sections 1–6).
Questions for authors
  • If the original question admits multiple answers and we only have one reference, one alternative way is to rewrite the question so that it strictly maps to the reference answer. How does this baseline compare with the proposed method, in terms of cost/performance, etc.?
  • How does the corrected reward perform in RL-trained models on QA benchmarks, compared to other baselines like A2Search[1]?

Reference

[1] Zhang, Fengji, et al. "A 2^ 2 Search: Ambiguity-Aware Question Answering with Reinforcement Learning." International Conference on Learning Representations. Vol. 2026. 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.