acceptodds
Under review as a conference paper at ICLR 2027

CAFE: Self-Improving Search Agents Need Co-Evolving Feedback

Abstract

Reliable search requires more than acquiring external evidence. An agent must also recognize and recover from errors as its trajectory unfolds. In-trajectory feed- back provides a mechanism for such recovery by diagnosing where the search has drifted and redirecting subsequent reasoning steps. This is particularly im- portant in long-horizon search, where an early directional error may receive no immediate corrective signal and can compound across later steps. Making such feedback learnable, however, creates a coupled problem: the agent must learn when to request and use feedback, while the critic must learn corrections from outcome-confounded rollouts as the agent’s failure patterns evolve. We introduce CAFE (Coupled Agent–Feedback Evolution), a framework in which a shared- parameter model alternates between search-agent and critic roles. CAFE initial- izes feedback-conditioned recovery from trajectories built around the base agent’s own failures, then couples online and offline optimization. During online RL, a comparative feedback estimate uses a prompt-level call–skip success gap to shape request returns, while feedback-aware advantage shaping reweights token advan- tages before and after feedback. Offline, rollout-derived preference optimization learns feedback from matched successful and unsuccessful trajectories. On seven agentic search benchmarks, CAFE outperforms the evaluated RL-based search agents on average, retains its gains across all six out-of-domain benchmarks, and reduces answer-level hallucinations. One-sided ablations show that improving only the agent or only the critic eventually plateaus, whereas alternating the two updates continues to improve performance. These findings suggest that a self- improving search agent needs feedback that co-evolves with the policy it guides.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large langugaes model, Agent,Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UNjWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAFE trains a shared model as a search agent and an on-demand trajectory critic. It begins with repaired failure trajectories, then alternates online reinforcement learning with offline preference learning for feedback generation. The online objective combines a call–skip success gap with advantage shaping around feedback requests; the offline stage learns from successful and unsuccessful feedback-assisted trajectories. The results suggest that this combination improves search performance.

Critical strengths
  1. Section 2.2 identifies a useful credit-assignment problem: a successful trajectory can contain a mistake followed by recovery. Equation (5) gives this distinction a concrete role in training by reducing positive credit on the prefix and increasing it on the continuation for eligible rollouts. Connecting an in-trajectory intervention to how the surrounding actions are reinforced is the paper's most interesting idea.

  2. The initialization in Section 2.1 fits that motivation well. Preserving the base agent's failed prefix and adding corrective feedback with a successful continuation exposes the model to recovery from states its own policy visits. This gives the request, critique, and continuation a common training context and provides a sensible starting point for the subsequent online learning.

  3. The main results are encouraging. Table 1 reports an increase from 49.7/58.0 to 52.5/60.7 average EM/F1 over GRPO, with gains on all seven benchmarks. Table 2 shows contributions from both CFE and advantage shaping, while Table 3 shows that RDPO improves over successful-feedback SFT in three of four configurations. Together, these results give substance to the proposed training design.

  4. Section 4.2 examines the interaction between the two roles directly. Figure 2a holds either the agent or critic fixed, and Figure 3 varies the critic while holding an agent checkpoint fixed. For the final agent, replacing the SFT critic with the final critic raises EM from 80.6 to 84.0. This is useful evidence that the learned critic contributes to task performance.

Critical weaknesses
  1. Equation (2) compares rollouts that chose different feedback routes and may have reached different search histories. Its success gap therefore mixes feedback effects with differences in the states where requests occur. Equation (3) then applies the same gap to every calling rollout for that prompt. This can provide a useful shaping reward, but it does not identify the value of a particular intervention. The closing interpretation in Section 2.2, that CFE determines when a request is worthwhile, needs this distinction.

  2. Section 2.3 prefers feedback from a successful trajectory over feedback from a failed one. Prefix matching and the semantic filter in Appendix A.4 make the comparison more informative, but subsequent success still depends on the agent's actions after the critique. RDPO consequently learns from feedback associated with recovery, without establishing which correction produced that recovery. The contribution is a practical way to learn from indirect feedback labels; the claim of reducing outcome confounding should be understood within that limitation.

  3. Section 2.3 states that RDPO updates the full shared model, so it can directly change the agent's search behavior as well as its feedback. The frozen-role and cross-play results in Section 4.2 support the usefulness of critic adaptation, but they do not make that adaptation the sole explanation for the alternating procedure's gains. The conclusion in Section 6 that a self-improving agent needs evolving feedback is therefore broader than the evidence for this particular coupled training framework.

Questions for authors
  1. In Section 2.2, should CFE be understood as a prompt-level reward heuristic or as an estimate of the value of an individual request? The interpretation affects what Equation (5)'s credit redistribution can be said to accomplish.

  2. Is the central claim in Sections 2.3 and 6 about keeping feedback supervision current with the agent's failures, or about joint adaptation within a shared model? Distinguishing these would sharpen the contribution.

5 comments
AuthorsOpenReview ↗

Response to Weakness1 & Weakness 2

Thank you immensely for your time and efforts, as well as the helpful and constructive feedback! Here, we give point-by-point responses to your comments.

Weakness 1& Question1 : CFE as a prompt-level reward-shaping heuristic


Thank you for highlighting this distinction. CFE should be understood as a prompt-level reward-shaping heuristic, rather than an estimate of the value of an individual feedback request. Equation (2) compares calling and skipping rollouts within the same prompt, holding the question fixed. However, the two groups may reach different search histories before choosing whether to request feedback. The resulting success gap therefore reflects both feedback effects and differences in the states in which feedback is requested. Equation (3) assigns this same prompt-level gap to all calling rollouts, without attributing value to any particular request.

To examine this pre-feedback imbalance, we measured whether the evidence retrieved by comparable decision points contained the gold answer. The gold answer was present in 12% of calling rollouts and 44% of skipping rollouts, a difference of 32 percentage points. This pattern is consistent with the agent requesting feedback more often when the evidence collected so far does not contain the answer. It also shows that the two groups differ before feedback is provided, so their subsequent success gap cannot be attributed solely to feedback.

Accordingly, CFE provides an aggregate signal for shaping feedback-seeking behavior within each prompt; it does not identify the causal benefit of a specific request at a particular state.

Weakness 2 : Fixed-prefix validation of the RDPO preference labels


Estimating the effect of feedback itself would require comparing continuations with and without feedback from the same prefix. We ran exactly the prefix-matching experiment described and draw a clear conclusion: the preferred feedback does lead to higher success.

Setup. We sampled 750 RDPO pairs from our 2Wiki run (150 from each of iterations 1–5). For each pair we froze the agent checkpoint and the trajectory up to the first feedback request, injected one feedback, and continued the rollout 8 times with the training sampling settings.

FeedbackOverall recovery rate
Chosen69.2%
Rejected53.0%
RDPO iterationChosenRejectedGap (percentage points)
163.5%54.7%8.8
266.7%53.8%12.9
366.9%51.7%15.3
474.2%50.5%23.7
574.7%54.4%20.3

Overall, chosen feedback yielded a recovery rate 16.2 percentage points higher than rejected feedback. Recovery with chosen feedback rose from 63.5% in iteration 1 to 74.7% in iteration 5. The gap peaked at 23.7 percentage points in iteration 4 and was 20.3 percentage points in iteration 5. The larger gaps in later iterations are consistent with RDPO training improving the critic’s ability to provide feedback that helps the agent recover.

We also asked DeepSeek V4.1 Flash to compare the two feedback responses in each of the same 750 RDPO pairs, with the original preference labels hidden.

Judge outcomeShare of pairs
Preferred chosen feedback64.67%
Tie2.67%
Preferred rejected feedback32.67%

The judge favored chosen feedback, consistent with the rollout results. The judge assessed the feedback itself, while the rollouts measured the agent’s subsequent performance. Together, the two evaluations provide complementary evidence that the RDPO preference labels identify feedback that is more useful to the agent.

AuthorsOpenReview ↗

Response to Weakness 3

Weakness 3: Interpreting the gains under shared-parameter training


Thank you for highlighting this distinction. RDPO directly optimizes feedback generation, but because the agent and critic share all parameters, its updates can also change the agent’s search and reasoning behavior. CAFE implements evolving feedback supervision through the joint adaptation of feedback generation and use within one model. The benefits of alternating training may therefore involve improved feedback, changes in agent capabilities, and interactions between the two.

Evidence from fixed-agent controls. Our experiments show that adapting the feedback model can improve downstream performance while the agent remains fixed. In the feedback-only control in Figure 2(a), freezing the SFT agent and updating only the model providing feedback increases the 2Wiki score, (EM+F1)/2(\mathrm{EM}+\mathrm{F1})/2, from 67.7 to 71.3. In the cross-play evaluation in Figure 3, holding the final agent fixed and replacing the SFT critic with the iteration-5 critic improves EM/F1 from 80.6/86.6 to 84.0/89.2. These controls support critic adaptation as one contributor to the observed improvements.

Comparison with additional online RL. We further examine the value of targeted feedback optimization relative to extending online training. Starting from the online RL configuration that uses both CFE and advantage shaping but no RDPO, we extend training from 500 to 507, 525, and 545 steps. Based on actual sequence lengths, the estimated model computation for five RDPO rounds is equivalent to approximately 6.72 online RL steps. Thus, 507 steps is the nearest integer-step budget match, with its seven additional steps consuming approximately 1.04 times the RDPO update budget.

The table reports equally weighted mean EM/F1 across seven benchmarks. Compute ratios refer to additional online RL computation relative to the estimated cost of all five RDPO updates, not to total training compute.

ConfigurationOnline RL stepsRDPO updatesExtra RL compute / five-round RDPO computeMean EMMean F1
Online RL only50000×51.9460.29
Extended online RL50701.04×51.3159.53
Extended online RL52503.7×51.8060.18
Extended online RL54506.7×52.2360.27
Full CAFE5005—52.5460.74

Full CAFE achieves higher mean EM and F1 than both the near-budget-matched control and the larger-budget controls. Even when additional online RL consumes approximately 6.7 times the RDPO update budget, the 545-step baseline remains below full CAFE. These results support allocating part of the training budget to feedback optimization in our setting, while remaining compatible with direct transfer to the agent through shared-parameter updates.

Evidence at the individual-feedback level. Our additional fixed-prefix experiment, described in the response to W2, holds the agent checkpoint and pre-feedback history fixed. Preferred feedback yields a recovery rate 16.2 percentage points higher than rejected feedback. Rejected feedback, in turn, yields a recovery rate 13.0 percentage points lower than non-informative placeholder feedback. This comparison shows that feedback can impede subsequent recovery and that RDPO preference labels provide a training signal for discouraging such harmful feedback. Together with the fixed-agent controls, these results support feedback adaptation as a concrete pathway through which CAFE can improve performance.

The intended emphasis of the title and Section 6 is that feedback mechanisms should adapt to changes in the agent’s capabilities and failure modes. Our fixed-agent controls support the value of updated feedback, while the alternating-training experiments support joint optimization of feedback generation and use. We will clarify in Section 6 that feedback improvement, direct changes in agent capabilities, and cross-role transfer may all contribute, and align the scope of our conclusions with the evidence from this coupled training framework.

AuthorsOpenReview ↗

Response to Question 2

Question 2: Evolving feedback supervision versus shared-model adaptation


Thank you for encouraging us to clarify these two levels. Our central claim concerns keeping feedback supervision aligned with the agent’s evolving capabilities and failure modes. CAFE refreshes this supervision by reconstructing feedback preference pairs from recent trajectories. Alternating optimization within a shared model is the concrete training framework we use to pursue this objective; parameter sharing itself is not a requirement of the underlying idea.

The distinction is also reflected in the training procedure. During online RL, feedback is generated using the current shared weights, but feedback tokens are excluded from the policy-gradient loss. Shared-parameter updates can therefore change feedback behavior even without explicit feedback supervision. RDPO is the only phase of the alternating procedure that directly supervises feedback generation.

Conversely, because the parameters are shared, RDPO can also directly affect the agent’s behavior. The gains of the full framework may thus arise from improved feedback, changes in agent capabilities, and their interaction. We attribute these gains to the coupled training procedure rather than exclusively to critic improvement, as discussed in our response to Weakness 3.

Our fixed-prefix evaluation provides evidence that the refreshed preference pairs are informative for the corresponding agent at each stage. Across all five iterations, we hold the agent checkpoint and pre-feedback history fixed and compare preferred and rejected feedback mined from recent trajectories. Preferred feedback yields recovery rates 8.8-23.7 percentage points higher than rejected feedback, with a positive difference at every iteration. These results support the usefulness of the stage-specific feedback supervision constructed by CAFE.


Thank you for your thorough review and insightful comments on our manuscript! We have carefully considered your suggestions and have made the revisions and clarifications to address the concerns raised. We would be more than willing to address any additional concerns you may have.

Reviewer_zufhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CAFE, an Agent–Critic co-evolutionary training framework that aims to continuously improve agent reasoning and search capabilities through alternating optimization of the Agent and Critic. Its main contribution lies in using Critic feedback to guide Agent improvement while optimizing the Critic through RDPO, allowing feedback quality to evolve alongside the Agent. Experiments on multiple search-based question-answering benchmarks, including BrowseComp-Plus, demonstrate the effectiveness of the proposed approach. Ablation studies, analyses of critic update frequency, and visualizations of the co-evolutionary process further support the framework's design. However, the reliability of RDPO preference labels and the independent contribution of co-evolutionary training beyond inference-time Agent–Critic interaction remain insufficiently validated.

Critical strengths

The paper proposes CAFE, a framework that enables Agent–Critic co-evolution through alternating optimization, rather than relying solely on a fixed feedback model. This design allows the Critic to continuously improve its feedback quality as the Agent evolves, thereby mitigating the limitations of static feedback mechanisms. The framework offers a potentially valuable direction for further exploration.

Critical weaknesses

Weakness 1: Outcome Confounding in RDPO Preference Labels. RDPO labels feedback from successful trajectories as chosen and feedback from failed trajectories as rejected. Although the authors use prefix matching and LLM filtering, these strategies do not fully address outcome confounding. Even with similar pre-feedback histories, subsequent search actions, retrieved evidence, and reasoning can differ substantially. Thus, final success or failure cannot be reliably attributed to feedback quality itself. A counterfactual evaluation that applies different feedback to the same pre-feedback state would provide stronger validation of the preference labels.

Weakness 2: Insufficient Baselines to Establish the Necessity of Co-Evolutionary Training. The paper lacks two important baselines to isolate the benefits of co-evolutionary training. First, a training-free Agent–Critic baseline, where the base model alternates between agent and critic roles through prompting alone, is needed to distinguish the gains from co-evolutionary training versus inference-time feedback interaction. Second, a compute-matched baseline that reallocates the resources spent on critic updates (RDPO) to additional agent training data or online RL steps is necessary to assess whether co-evolving feedback is more effective than simply scaling agent training. Without these comparisons, it remains unclear whether co-evolving feedback is truly necessary or provides advantages beyond simpler alternatives.

Questions for authors

Question 1: Generalization to More Challenging Benchmarks. The evaluation primarily focuses on standard SearchQA benchmarks and BrowseComp-Plus. Have the authors considered evaluating CAFE on more challenging knowledge-intensive benchmarks, such as HLE and GPQA, to further assess its generalization to difficult reasoning tasks?

Question 2: Critic Update Frequency and Effectiveness. Table 6 explores different critic update frequencies, while Figures 2–3 demonstrate the benefits of critic improvement and agent–critic co-evolution. However, how was the critic update frequency selected, and could an adaptive update schedule be more effective as the agent evolves? Additionally, can the authors provide a more direct evaluation of critic quality and quantify how improvements in the critic translate into downstream agent performance?

Question 3: Alternative Co-Evolution Strategies. CAFE focuses on co-evolving a search agent and its feedback critic. An alternative would be to co-evolve the search agent with a task generator that continuously produces challenging search problems tailored to the agent's current capabilities. Have the authors considered this alternative? Could adaptive task generation provide comparable or greater improvements than feedback co-evolution under the same training budget?

8 comments
AuthorsOpenReview ↗

Response to Weakness 1

Thank you immensely for your active engagement and instructive comments ! We have carefully considered your suggestions and have made the following revisions and clarifications to address the concerns raised.

Weakness 1 : Fixed-prefix validation of the RDPO preference labels


We thank the reviewer for this concrete suggestion. To examine whether the call and skip groups differ before feedback is given, we measured whether the evidence retrieved by comparable decision points contained the gold answer. The evidence contained the gold answer in 12% of calling rollouts and 44% of skipping rollouts, a gap of 32 percentage points. This pattern suggests that the agent tends to request feedback when the evidence retrieved so far does not contain the answer. It also highlights why we interpret the call–skip success gap in Eq. (2) as a prompt-level reward-shaping signal. Estimating the effect of feedback itself would require comparing continuations with and without feedback from the same prefix. We ran exactly the prefix-matching experiment described and draw a clear conclusion: the preferred feedback does lead to higher success.

Setup. We sampled 750 RDPO pairs from our 2Wiki run (150 from each of iterations 1–5). For each pair we froze the agent checkpoint and the trajectory up to the first feedback request, injected one feedback, and continued the rollout 8 times with the training sampling settings.

FeedbackOverall recovery rate
Chosen69.2%
Rejected53.0%
RDPO iterationChosenRejectedGap (percentage points)
163.5%54.7%8.8
266.7%53.8%12.9
366.9%51.7%15.3
474.2%50.5%23.7
574.7%54.4%20.3

Overall, chosen feedback yielded a recovery rate 16.2 percentage points higher than rejected feedback. Recovery with chosen feedback rose from 63.5% in iteration 1 to 74.7% in iteration 5. The gap peaked at 23.7 percentage points in iteration 4 and was 20.3 percentage points in iteration 5. The larger gaps in later iterations are consistent with RDPO training improving the critic’s ability to provide feedback that helps the agent recover.

Judge evaluation. We also asked DeepSeek V4.1 Flash to compare the two feedback responses in each of the same 750 RDPO pairs, with the original preference labels hidden.

Judge outcomeShare of pairs
Preferred chosen feedback64.67%
Tie2.67%
Preferred rejected feedback32.67%

The judge favored chosen feedback, consistent with the rollout results. The judge assessed the feedback itself, while the rollouts measured the agent’s subsequent performance. Together, the two evaluations provide complementary evidence that the RDPO preference labels identify feedback that is more useful to the agent.

AuthorsOpenReview ↗

Response to Weakness 2 (1/2)

Weakness 2: Insufficient Baselines


Thank you for suggesting these two comparisons. We clarify the training-free Agent–Critic baseline already included in Table 1 and add online RL baselines that reallocate the estimated RDPO update budget to additional agent training.

A. Training-free Agent–Critic baseline

The Qwen2.5-7B row under CAFE in Table 1 corresponds to the training-free Agent–Critic baseline: the base model serves as both agent and critic through prompting, using the CAFE interaction protocol without additional parameter updates. To make this comparison explicit, we additionally evaluate the base model with direct answering.

Each entry reports EM/F1 (%), and Avg. is the equally weighted average across the seven benchmarks.

Configuration2WikiHotpotQAMuSiQuePopQATriviaQABamboogleNQAvg.
Qwen2.5-7B-Instruct, direct answering32.4/39.9822.4/30.726.2/14.2822.8/29.4945.8/54.8828.0/40.8819.4/29.9925.29/34.32
Training-free Agent–Critic44.8/54.441.8/53.620.4/28.834.4/42.156.4/65.637.6/49.331.6/41.338.14/47.87

The inference-time search-and-feedback framework improves average EM/F1 by 12.86/13.55 percentage points over direct answering, without additional training. Full CAFE subsequently reaches 52.54/60.74 after training. These comparisons distinguish the gains of the inference-time framework from the further gains obtained through training.

B. Reallocating the RDPO update budget to additional online RL

We start from the online RL configuration in Table 2 that uses both CFE and advantage shaping but no RDPO. We extend this configuration from 500 steps to 507, 525, and 545 steps, covering a near-matched budget and two larger budgets.

Compute accounting. We estimate model FLOPs from the actual sequence lengths. For Qwen2.5-7B, the number of parameters involved in matrix multiplications is

Nmm=28(2⋅35842+2⋅3584⋅512+3⋅3584⋅18944)+152064⋅3584≈7.070×109.N_{\mathrm{mm}}=28\left(2\cdot3584^2+2\cdot3584\cdot512+3\cdot3584\cdot18944\right)+152064\cdot3584\approx7.070\times10^9.

For a sequence of length LL, we approximate forward computation as F(L)=2NmmL+2nlayerdL2F(L)=2N_{\mathrm{mm}}L+2n_{\mathrm{layer}}dL^2, where nlayer=28n_{\mathrm{layer}}=28 and d=3584d=3584. The second term accounts for causal attention. Both training procedures use gradient checkpointing, so a policy-training pass is estimated as 4F(L)4F(L), including the forward pass, backward pass, and forward recomputation.

Each online RL step contains 128×8=1,024128\times8=1{,}024 trajectories. Its compute includes rollout generation, old-policy log-probability evaluation (1F1F), reference-model log-probability evaluation (1F1F), and the actor update (4F4F). Generation is accounted for with shared prompt prefilling within each rollout group. Actual rollout lengths from steps 501–520 yield approximately 2.779 million tokens per step, close to the 2.803 million recorded in the training logs, giving

CRL≈2.765×1017 FLOPs per step.C_{\mathrm{RL}}\approx2.765\times10^{17}\ \text{FLOPs per step}.

For RDPO, each preference pair contains a chosen and a rejected sequence. Each sequence incurs 4F4F for policy training and 1F1F for reference-model evaluation. Using the cap of 2,000 pairs per round, one epoch, and the observed paired-sequence length distributions gives the following estimates:

RDPO roundMean total tokens per pairEstimated FLOPs
12,7864.024×10174.024\times10^{17}
22,7333.947×10173.947\times10^{17}
32,4933.592×10173.592\times10^{17}
42,4433.518×10173.518\times10^{17}
52,4383.510×10173.510\times10^{17}
TotalApproximately 25.8M tokens across all pairs1.859×10181.859\times10^{18}

The five RDPO rounds therefore consume approximately 6.72 online RL steps’ worth of model computation, or 1.33% of the estimated combined cost of 500 online RL steps and five RDPO rounds.

AuthorsOpenReview ↗

Reponse to Weakness 2 (2/2)

Budget-matched and larger-budget controls. The continuous budget-equivalent target is 500+CRDPO/CRL≈506.72500+C_{\mathrm{RDPO}}/C_{\mathrm{RL}}\approx506.72 RL steps. We therefore evaluate 507 steps, the nearest integer-step match. Its seven additional steps consume approximately 1.04 times the RDPO update budget, slightly exceeding the target. The 525- and 545-step controls allocate approximately 3.7 and 6.7 times that budget to additional online RL. These multipliers refer to the additional RL computation relative to RDPO, not to total training compute.

Configuration2WikiHotpotQAMuSiQuePopQATriviaQABamboogleNQAvg.
Online RL, 500 steps83.4/88.353.0/63.931.8/40.546.2/50.861.2/69.248.0/60.540.0/48.851.94/60.29
Online RL, 507 steps (1.04×)82.8/88.1552.8/63.0627.6/38.946.6/50.6761.8/68.7748.0/58.1739.6/49.0151.31/59.53
Online RL, 525 steps (3.7×)84.2/88.1552.8/64.4528.8/39.2547.2/51.6461.2/69.1949.6/60.3238.8/48.2451.80/60.18
Online RL, 545 steps (6.7×)83.8/88.4652.0/64.0230.4/39.7546.2/49.9960.8/68.3852.8/62.5639.6/48.7352.23/60.27
Full CAFE, 500 RL steps + RDPO84.0/89.253.4/64.830.2/39.146.4/51.662.6/69.950.4/61.440.8/49.252.54/60.74

Full CAFE exceeds the near-budget-matched 507-step baseline by 1.23 EM and 1.21 F1 percentage points. Additional online RL improves performance on some individual benchmarks, but all three extended baselines remain below full CAFE in average EM and F1. Even with approximately 6.7 times the RDPO update budget allocated to additional RL, the 545-step baseline remains 0.31 EM and 0.47 F1 points lower.

These results support the utility of RDPO within our coupled training framework: under the evaluated setting, its observed gains are not reproduced by simply allocating an equivalent or larger update budget to additional online RL.

Reviewer_nPdLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how a search agent can learn to request and use corrective feedback while also improving the feedback it generates. CAFE uses one shared model for the agent and critic roles. It starts with teacher-repaired trajectories that preserve the base agent's failed prefixes, then alternates online reinforcement learning with offline feedback preference optimization. The online update adds a prompt-level success gap between feedback-calling and feedback-skipping rollouts and adjusts credit before and after the first request. The offline update constructs feedback preferences from successful and unsuccessful rollouts with similar prefixes. Experiments on seven SearchQA benchmarks, using Qwen2.5 models at 3B and 7B, report improvements over outcome-only GRPO; a further 7B evaluation uses BrowseComp-Plus. Component ablations and agent–critic cross-play examine the training design, and the paper also reports fewer answers unsupported by retrieved evidence.

Critical strengths
  1. Preserving the base agent's failed prefixes is a sensible way to initialize recovery. The model learns to continue from mistakes it actually makes. Making feedback requests part of the action space and separating credit before and after a request fit this training setup.

  2. CAFE improves on the shared-initialization GRPO baseline on all seven benchmarks in Table 1. The same pattern holds at 3B in Table 4, so the improvement is not confined to the in-domain 2Wiki task or the 7B model.

  3. The ablations cover the main training choices. Table 2 separates CFE and advantage shaping, Table 3 compares RDPO with successful-feedback SFT under the same iteration schedule, and Figure 3 checks different agent–critic checkpoint combinations.

Critical weaknesses
  1. The call–skip gap in Equation (2) is computed from trajectories that choose their own route. Calling trajectories may therefore differ from skipping trajectories before feedback is given. For RDPO, prefix matching and GPT-5.1 filtering (Appendix A.4) reduce this mismatch, but the success labels still come from different continuations, and rejected feedback is placed under the successful rollout's history. A fixed-prefix comparison would help establish whether the feedback labels track better recovery.

  2. I could not find run-to-run variation or paired confidence intervals for Tables 1–3 and Figure 3. Some of the differences supporting stage-specific alignment are small. In Figure 3(b), for example, the iteration-4 agent scores 87.5 F1 with its matching critic and 87.4 with the SFT critic. Without uncertainty estimates, it is hard to tell whether this matching advantage is reliable.

  3. Section 4.4 defines hallucination, and Appendix D.2 gives the per-dataset rates, but the labeling procedure is not described. Who judged whether an answer was supported by the retrieved evidence, and under what instructions? The handling of invalid or missing answers is also unclear. This makes the hallucination result difficult to assess independently.

Questions for authors
  1. If available, could you report a fixed-prefix comparison of the preferred and rejected feedback from a small set of RDPO pairs, using the same agent? Does the preferred feedback lead to a higher success rate? Results from continuations without feedback would also help check the CFE signal.

  2. How many independent training runs were used for the main results and the cross-play matrix? Please report paired confidence intervals for the main gains and the critic comparisons, if available. Were the checkpoints and the 100 × 5 schedule chosen on a separate development set?

  3. Please describe how the hallucination labels in Figure 4(a) and Table 5 were produced: the annotator or judge, the instructions, the number of evaluated answers, and the treatment of invalid or missing outputs. If you used an automatic judge, please include any available manual checks of its labels.

6 comments
AuthorsOpenReview ↗

Response to Weakness 1 & Question1

Weakness 1& Question1 : Fixed-prefix validation of the RDPO preference labels


We thank the reviewer for this concrete suggestion. To examine whether the call and skip groups differ before feedback is given, we measured whether the evidence retrieved by comparable decision points contained the gold answer. The evidence contained the gold answer in 12% of calling rollouts and 44% of skipping rollouts, a gap of 32 percentage points. This pattern suggests that the agent tends to request feedback when the evidence retrieved so far does not contain the answer. It also highlights why we interpret the call–skip success gap in Eq. (2) as a prompt-level reward-shaping signal. Estimating the effect of feedback itself would require comparing continuations with and without feedback from the same prefix. We ran exactly the prefix-matching experiment described and draw a clear conclusion: the preferred feedback does lead to higher success.

Setup. We sampled 750 RDPO pairs from our 2Wiki run (150 from each of iterations 1–5). For each pair we froze the agent checkpoint and the trajectory up to the first feedback request, injected one feedback, and continued the rollout 8 times with the training sampling settings.

FeedbackOverall recovery rate
Chosen69.2%
Rejected53.0%
RDPO iterationChosenRejectedGap (percentage points)
163.5%54.7%8.8
266.7%53.8%12.9
366.9%51.7%15.3
474.2%50.5%23.7
574.7%54.4%20.3

Overall, chosen feedback yielded a recovery rate 16.2 percentage points higher than rejected feedback. Recovery with chosen feedback rose from 63.5% in iteration 1 to 74.7% in iteration 5. The gap peaked at 23.7 percentage points in iteration 4 and was 20.3 percentage points in iteration 5. The larger gaps in later iterations are consistent with RDPO training improving the critic’s ability to provide feedback that helps the agent recover.

Judge evaluation. We also asked DeepSeek V4.1 Flash to compare the two feedback responses in each of the same 750 RDPO pairs, with the original preference labels hidden.

Judge outcomeShare of pairs
Preferred chosen feedback64.67%
Tie2.67%
Preferred rejected feedback32.67%

The judge favored chosen feedback, consistent with the rollout results. The judge assessed the feedback itself, while the rollouts measured the agent’s subsequent performance. Together, the two evaluations provide complementary evidence that the RDPO preference labels identify feedback that is more useful to the agent.

AuthorsOpenReview ↗

Response to Weakness2 & Question 2

Weakness2 & Question 2 : Variation across training seeds and evaluation protocol


Thank you for requesting additional evidence on the robustness of our results. A complete CAFE training run comprises five rounds of alternating online RL and RDPO updates and takes approximately 2–3 days. Within the computational budget available during rebuttal phase, we prioritized additional independent training runs for full CAFE and now report results across three training seeds. Each entry reports EM/F1, and the final column is the equally weighted average across seven benchmarks.

Each entry reports EM/F1 (%). Avg. is the equally weighted average across the seven benchmarks.

Configuration2WikiHotpotQAMuSiQuePopQATriviaQABamboogleNQAvg.
Qwen2.5-7B-Instruct44.8/54.441.8/53.620.4/28.834.4/42.156.4/65.637.6/49.331.6/41.338.1/47.9
+ SFT63.1/72.444.2/55.019.0/27.535.2/42.152.6/62.542.4/51.828.8/39.140.8/50.1
+ SFT + GRPO80.6/86.650.8/61.027.2/36.644.6/49.060.4/67.945.6/56.238.4/48.649.7/58.0
+ SFT + CAFE (seed 1)84.0/89.253.4/64.830.2/39.146.4/51.662.6/69.950.4/61.440.8/49.252.5/60.7
+ SFT + CAFE (seed 2)83.6/88.252.4/63.230.6/38.547.2/51.466.0/72.549.6/60.840.2/49.552.8/60.6
+ SFT + CAFE (seed 3)84.0/89.153.0/63.931.8/40.546.4/50.763.8/71.248.0/59.839.8/50.152.4/60.8

Full CAFE achieves a seven-benchmark macro-average of 52.58 ± 0.20 EM and 60.70 ± 0.10 F1 (mean ± sample standard deviation across three training seeds). All three runs outperform the reported SFT + GRPO baseline in average EM and F1, with gains averaged across seeds of approximately 2.9 EM and 2.7 F1 percentage points. The overall improvement is therefore reproduced across all three training seeds under our experimental setting.

We agree that the iteration-4 comparison is a near tie. However, the overall cross-play pattern is consistent with our hypothesis that critic effectiveness can vary with the agent’s training stage. For instance, with the iteration-5 agent held fixed, the matching critic scored 89.2 F1 compared with 86.6 for the SFT critic. We also clarify that we established a separate development set for RL training. Both checkpoint selection and selection of the training schedule were based on development-set performance.

For Tables 2–3, each ablation configuration reports the mean of three evaluations of its corresponding checkpoint. Likewise, each agent–critic pairing in Figure 3 was evaluated three times, and the reported score is their mean. Repeated evaluation reduces inference-sampling noise, while the three independent CAFE training runs quantify variation across training seeds.

AuthorsOpenReview ↗

Reponse to Weakness 3 & Question 3

Weakness 3 & Question 3: Hallucination evaluation


Figure 4(a) reports an answer-level hallucination proxy based on answer occurrence in tool outputs. For each search trajectory, we check whether the reference answer and the model’s final answer appear in the tool outputs collected along that trajectory. An incorrect response (score = 0) is labeled as hallucinated if its final answer does not appear in these outputs.

This criterion covers two cases: (i) the reference answer was retrieved, but the model produced an incorrect answer absent from the tool outputs; and (ii) the reference answer was not retrieved, and the model nevertheless produced an incorrect answer absent from the tool outputs. If an incorrect answer does appear in the tool outputs, we categorize it as a retrieval or candidate-selection error rather than a hallucination under this metric.

Invalid outputs, outputs with no final answer, and explicit refusals are excluded from the analysis. For each dataset, the hallucination rate is the number of responses labeled as hallucinated divided by the total number of evaluated samples remaining after these exclusions, with each response counted at most once.

This metric uses answer occurrence as a proxy for retrieval support, rather than assessing semantic support for individual factual claims. An answer’s presence in the tool outputs does not necessarily mean that the surrounding evidence supports it.


Thank you for your thorough review and insightful comments on our manuscript! We have carefully considered your suggestions and have made the revisions and clarifications to address the concerns raised.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you & Summary of the Discussion Phase

Dear AC and Reviewers,

We sincerely thank the Area Chair for overseeing the review of our submission and for the time and care devoted to the discussion. We are also deeply grateful to all three reviewers for their thoughtful, constructive feedback and their engagement, which have substantially strengthened the paper. We briefly summarize the discussion below.

Acknowledged strengths. Reviewers found the design well motivated, describing the failed-prefix initialization as "a sensible way to initialize recovery" (nPdL) and feedback-aware credit assignment as "the paper's most interesting idea" (UNjW). They viewed agent–critic co-evolution as "a potentially valuable direction" (zufh) and noted consistent gains over GRPO on all seven benchmarks (nPdL, UNjW), supported by thorough ablations (nPdL, zufh, UNjW).

Main concerns and responses.

  • Confounding in the RDPO labels (all reviewers). In a fixed-prefix evaluation on 750 pairs, chosen feedback yields 16.2 points higher recovery than rejected feedback, with a positive gap at every iteration. A blind judge also agrees with 64.7% of the labels. Reviewer UNjW confirmed that this "directly addresses" the concern.
  • Interpretation of CFE (nPdL, UNjW). We clarified CFE as a prompt-level shaping statistic, supported by an analysis of the pre-feedback imbalance between calling and skipping rollouts.
  • Baselines and compute (zufh, UNjW). The training-free agent–critic baseline is the base-model row of Table 1. Reallocating up to 6.7× the RDPO compute to additional online RL still falls short of full CAFE.
  • Robustness and evaluation (nPdL, zufh). Although each full CAFE training run takes several days, we made our best effort within the rebuttal period to train three independent seeds, and every run consistently outperforms the GRPO baseline. We also evaluated CAFE on the more challenging GPQA-Diamond and HLE benchmarks, where it improves over both the base model and the SFT checkpoint, and we clarified the hallucination evaluation criterion. We believe these additions address the reviewers' concerns about robustness and generalization.

Revisions. The revised PDF clarifies CFE in Section 2.2 and adds Appendix E with the attribution and compute-matched analyses. The remaining results, the hallucination criterion, and the narrowed scope of the conclusion will be incorporated into the final version.

We are encouraged by the positive feedback from all three reviewers, and we are grateful that Reviewer UNjW raised the rating after the discussion, noting that our responses "substantially address the concerns". We are truly grateful to the Area Chair for handling our submission and for taking the time to consider this summary. We also thank all reviewers once more for their generous time and insightful comments, and we would be glad to provide any further clarification.

Sincerely, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.