acceptodds
Desk rejected at ICLR 2027

ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM-based search agents are trained predominantly with outcome-only reward, leaving the search process itself unsupervised. This signal degenerates on outcome-homogeneous groups where all sampled trajectories share the same correctness, yielding zero within-group advantage and no gradient. Existing process supervision either trains a costly scoring model or generates per-query rubrics that are inconsistent across queries. We propose **ARBOR** (**A**daptive **R**ubric **B**uffer for **O**nline **R**eward), a reusable process-reward framework that maintains a rubric memory shared across queries. Query-local drafts induced from contrastive trajectories are admitted, consolidated into cross-query common rubrics, and retired as the policy evolves. A small active subset of common rubrics scores trajectories via sparse pairwise judging, and the resulting scores are added to the base reward, providing process-level gradient even when outcome reward is uniform. ARBOR consistently outperforms GRPO and DAPO baselines on four multi-hop QA benchmarks, raising average LLM-judge accuracy by up to **4.2** points and converting up to **42%** of otherwise-zero-gradient training groups into informative ones.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Search Agents, Multi-hop Question Answering, Process Reward, Rubric Memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wqjhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ARBOR, a memory framework for metrics. The intuition is straightforward: trajectories with identical outcomes (e.g. correctness) can have other differences, which could be further fine-grained using this framework.

Critical strengths
  1. ARBOR shows consistent performance improvement in 3 sizes and across many tasks.
  2. The idea of having an online metrics framework is quite unique.
Critical weaknesses
  1. The necessity of having an online metric framework is not strictly proven. A high percentage of groups are outcome-homogeneous groups, for which the induction stage will be skipped. Does this mean that, without memory (in Figure 3), there will be no rubrics? In Appendix E.1 the author write Ideal Search Agent Vision, can the author use that as metrics and run the experiment to see if online update are necessary. Some of the top-5 metrics created by ARBOR is quite similar to (overlapped with) the Ideal Search Agent Vision.
  2. The training only goes for 60 steps, and each rollout only contains 8 rollouts. Does ARBOR work better with low rollout counts, as there will be a much low chances of a group of trajectories are either all correct/wrong if the rollout counts are higher. Also, does having more training step address this issue? as ARBOR may provide some additional gradients in the early steps, but may not give better results when the training is sufficient, or it could also give misleading rubrics.
Questions for authors

See Weakness.

Reviewer_3doqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ARBOR, a framework for training search agents with an evolving, reusable rubric buffer. It induces rubrics from contrasting trajectories from the same prompt group, consolidates them into shared criteria, and uses pairwise judgments to augment outcome rewards. This method can provide learning signals when all trajectories in a prompt group receive identical outcome rewards. Experiments across different model sizes and multi-hop QA benchmarks show improvements over GRPO and DAPO.

Critical strengths
  • Clear target: The paper focuses on the issue of learning signal degeneration in outcome-homogeneous rollout groups in group-relative RL training.
  • Consistent empirical gains: ARBOR improves average performance over GRPO and DAPO across three model sizes and four multi-hop QA benchmarks.
Critical weaknesses
  • Insufficient comparison with existing evolving-rubric methods. This paper lack comparisons with closely related methods such as DR. Tulu [1] and RLER [2], which already proposes the idea of generating rubrics from contrasting rollouts, maintaining a dynamic rubric buffer, and removing uninformative rubrics during training. I think these overlaps make it critical to clarify ARBOR's incremental contribution beyond existing rubric evolution mechanisms. Although the paper includes a "w/o memory" ablation, this variant does not maintain a persistent query-specific buffer and thus cannot substitute for an evolving-rubric baseline. A controlled comparison would help establish the additional value of cross-query consolidation and sharing.
  • The motivation concerning cross-query rubric conflicts is insufficiently substantiated. The paper mentions "the process criteria should be general, cross-query reusable standards, rather than query-specific rubrics that may conflict across queries" in the Introduction. However, I am a bit confused about this claim. Different queries may legitimately require different behaviors, such as additional verification versus early termination. Such task-dependent differences should be distinguished from inconsistent evaluation standards. Examples or quantitative analyses showing that harmful conflicts occur, and that the shared buffer mitigates them, would strengthen this motivation.
  • The effectiveness and efficiency of contrastive anchor selection is not fully evaluated. Section 3.2 uses two contrast pairs, best--worst and best--hard-negative, to reduce the number of comparisons relative to exhaustive pairing. However, the experiments do not compare this strategy against exhaustive pairing, random pairing, or either pair alone. Consequently, it remains unclear whether the reduced comparison set sacrifices rubric coverage or quality, and whether both contrast pairs are necessary.
  • The robustness of F1-based anchor selection remains only partially examined. Section 4.5 compares F1 anchors with MC-based anchors and finds no improvement from the latter, providing some support for the selected choice. However, this does not address potential discrepancies between lexical F1 and semantic correctness, which could lead to misleading anchor selection. Since the framework already adopts an external LLM-Judge for rubric induction, a comparison with LLM-based anchor selection would be informative.

[1] Shao, Ru-Lin et al. “DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research.”

[2] Sheng, Leheng et al. “Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubrics.”

Typos:

  • Page 2, Figure 1: 46-57 miles fits the “approximately 45 miles.
  • Page 5, L232-233: checked at admission stage
Questions for authors

Please refer to the Weaknesses.

Reviewer_LG3NOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper focuses on the RL optimization of search agents using process reward. In more detail, the paper introduces ARBOR, a reusable process-reward framework that continuously learns, maintains, and applies process criteria to improve trajectory-level reward signals. ARBOR maintains two rubric pools: a candidate pool for query-specific rubrics derived from contrastive trajectories and a common pool for reusable criteria shared across queries. ARBOR is evaluated on four multihop QA benchmarks.

Critical strengths
  • The paper focuses on optimization of search agents which is an important and timely topic.
  • Process reward is important and interesting.
  • The results compared to the baselines are encouraging.
Critical weaknesses
  • There are numerous process reward methods these days for search agents. With a quick search I found 20+ papers and none was used and compared with the proposed solution. Some reported stronger results that those reported in this paper. Therefore, I think the paper does not demonstrate the full understanding of the literature.

  • The experiments are only evaluated on multihop QA methods. Results on single-hop QA (NQ, HotpotQA, etc) or deep research benchmarks (BrowseCompPlus, Total Recall QA, etc) are unknown.

  • The proposed method seems to lack theoretical rigor and it's more or less an engineering effort without much theoretical exploration of why this approach works.

Questions for authors

None.

Comments on the paper

1 comment

Thanks for the reply. All my concerns have been resolved. I have increased my score to 3-weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.