- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CAFE trains a shared model as a search agent and an on-demand trajectory critic. It begins with repaired failure trajectories, then alternates online reinforcement learning with offline preference learning for feedback generation. The online objective combines a call–skip success gap with advantage shaping around feedback requests; the offline stage learns from successful and unsuccessful feedback-assisted trajectories. The results suggest that this combination improves search performance.
Critical strengths
-
Section 2.2 identifies a useful credit-assignment problem: a successful trajectory can contain a mistake followed by recovery. Equation (5) gives this distinction a concrete role in training by reducing positive credit on the prefix and increasing it on the continuation for eligible rollouts. Connecting an in-trajectory intervention to how the surrounding actions are reinforced is the paper's most interesting idea.
-
The initialization in Section 2.1 fits that motivation well. Preserving the base agent's failed prefix and adding corrective feedback with a successful continuation exposes the model to recovery from states its own policy visits. This gives the request, critique, and continuation a common training context and provides a sensible starting point for the subsequent online learning.
-
The main results are encouraging. Table 1 reports an increase from 49.7/58.0 to 52.5/60.7 average EM/F1 over GRPO, with gains on all seven benchmarks. Table 2 shows contributions from both CFE and advantage shaping, while Table 3 shows that RDPO improves over successful-feedback SFT in three of four configurations. Together, these results give substance to the proposed training design.
-
Section 4.2 examines the interaction between the two roles directly. Figure 2a holds either the agent or critic fixed, and Figure 3 varies the critic while holding an agent checkpoint fixed. For the final agent, replacing the SFT critic with the final critic raises EM from 80.6 to 84.0. This is useful evidence that the learned critic contributes to task performance.
Critical weaknesses
-
Equation (2) compares rollouts that chose different feedback routes and may have reached different search histories. Its success gap therefore mixes feedback effects with differences in the states where requests occur. Equation (3) then applies the same gap to every calling rollout for that prompt. This can provide a useful shaping reward, but it does not identify the value of a particular intervention. The closing interpretation in Section 2.2, that CFE determines when a request is worthwhile, needs this distinction.
-
Section 2.3 prefers feedback from a successful trajectory over feedback from a failed one. Prefix matching and the semantic filter in Appendix A.4 make the comparison more informative, but subsequent success still depends on the agent's actions after the critique. RDPO consequently learns from feedback associated with recovery, without establishing which correction produced that recovery. The contribution is a practical way to learn from indirect feedback labels; the claim of reducing outcome confounding should be understood within that limitation.
-
Section 2.3 states that RDPO updates the full shared model, so it can directly change the agent's search behavior as well as its feedback. The frozen-role and cross-play results in Section 4.2 support the usefulness of critic adaptation, but they do not make that adaptation the sole explanation for the alternating procedure's gains. The conclusion in Section 6 that a self-improving agent needs evolving feedback is therefore broader than the evidence for this particular coupled training framework.
Questions for authors
-
In Section 2.2, should CFE be understood as a prompt-level reward heuristic or as an estimate of the value of an individual request? The interpretation affects what Equation (5)'s credit redistribution can be said to accomplish.
-
Is the central claim in Sections 2.3 and 6 about keeping feedback supervision current with the agent's failures, or about joint adaptation within a shared model? Distinguishing these would sharpen the contribution.
5 comments
Response to Weakness 3
Weakness 3: Interpreting the gains under shared-parameter training
Thank you for highlighting this distinction. RDPO directly optimizes feedback generation, but because the agent and critic share all parameters, its updates can also change the agent’s search and reasoning behavior. CAFE implements evolving feedback supervision through the joint adaptation of feedback generation and use within one model. The benefits of alternating training may therefore involve improved feedback, changes in agent capabilities, and interactions between the two.
Evidence from fixed-agent controls. Our experiments show that adapting the feedback model can improve downstream performance while the agent remains fixed. In the feedback-only control in Figure 2(a), freezing the SFT agent and updating only the model providing feedback increases the 2Wiki score, , from 67.7 to 71.3. In the cross-play evaluation in Figure 3, holding the final agent fixed and replacing the SFT critic with the iteration-5 critic improves EM/F1 from 80.6/86.6 to 84.0/89.2. These controls support critic adaptation as one contributor to the observed improvements.
Comparison with additional online RL. We further examine the value of targeted feedback optimization relative to extending online training. Starting from the online RL configuration that uses both CFE and advantage shaping but no RDPO, we extend training from 500 to 507, 525, and 545 steps. Based on actual sequence lengths, the estimated model computation for five RDPO rounds is equivalent to approximately 6.72 online RL steps. Thus, 507 steps is the nearest integer-step budget match, with its seven additional steps consuming approximately 1.04 times the RDPO update budget.
The table reports equally weighted mean EM/F1 across seven benchmarks. Compute ratios refer to additional online RL computation relative to the estimated cost of all five RDPO updates, not to total training compute.
| Configuration | Online RL steps | RDPO updates | Extra RL compute / five-round RDPO compute | Mean EM | Mean F1 |
|---|---|---|---|---|---|
| Online RL only | 500 | 0 | 0× | 51.94 | 60.29 |
| Extended online RL | 507 | 0 | 1.04× | 51.31 | 59.53 |
| Extended online RL | 525 | 0 | 3.7× | 51.80 | 60.18 |
| Extended online RL | 545 | 0 | 6.7× | 52.23 | 60.27 |
| Full CAFE | 500 | 5 | — | 52.54 | 60.74 |
Full CAFE achieves higher mean EM and F1 than both the near-budget-matched control and the larger-budget controls. Even when additional online RL consumes approximately 6.7 times the RDPO update budget, the 545-step baseline remains below full CAFE. These results support allocating part of the training budget to feedback optimization in our setting, while remaining compatible with direct transfer to the agent through shared-parameter updates.
Evidence at the individual-feedback level. Our additional fixed-prefix experiment, described in the response to W2, holds the agent checkpoint and pre-feedback history fixed. Preferred feedback yields a recovery rate 16.2 percentage points higher than rejected feedback. Rejected feedback, in turn, yields a recovery rate 13.0 percentage points lower than non-informative placeholder feedback. This comparison shows that feedback can impede subsequent recovery and that RDPO preference labels provide a training signal for discouraging such harmful feedback. Together with the fixed-agent controls, these results support feedback adaptation as a concrete pathway through which CAFE can improve performance.
The intended emphasis of the title and Section 6 is that feedback mechanisms should adapt to changes in the agent’s capabilities and failure modes. Our fixed-agent controls support the value of updated feedback, while the alternating-training experiments support joint optimization of feedback generation and use. We will clarify in Section 6 that feedback improvement, direct changes in agent capabilities, and cross-role transfer may all contribute, and align the scope of our conclusions with the evidence from this coupled training framework.
Response to Question 2
Question 2: Evolving feedback supervision versus shared-model adaptation
Thank you for encouraging us to clarify these two levels. Our central claim concerns keeping feedback supervision aligned with the agent’s evolving capabilities and failure modes. CAFE refreshes this supervision by reconstructing feedback preference pairs from recent trajectories. Alternating optimization within a shared model is the concrete training framework we use to pursue this objective; parameter sharing itself is not a requirement of the underlying idea.
The distinction is also reflected in the training procedure. During online RL, feedback is generated using the current shared weights, but feedback tokens are excluded from the policy-gradient loss. Shared-parameter updates can therefore change feedback behavior even without explicit feedback supervision. RDPO is the only phase of the alternating procedure that directly supervises feedback generation.
Conversely, because the parameters are shared, RDPO can also directly affect the agent’s behavior. The gains of the full framework may thus arise from improved feedback, changes in agent capabilities, and their interaction. We attribute these gains to the coupled training procedure rather than exclusively to critic improvement, as discussed in our response to Weakness 3.
Our fixed-prefix evaluation provides evidence that the refreshed preference pairs are informative for the corresponding agent at each stage. Across all five iterations, we hold the agent checkpoint and pre-feedback history fixed and compare preferred and rejected feedback mined from recent trajectories. Preferred feedback yields recovery rates 8.8-23.7 percentage points higher than rejected feedback, with a positive difference at every iteration. These results support the usefulness of the stage-specific feedback supervision constructed by CAFE.
Thank you for your thorough review and insightful comments on our manuscript! We have carefully considered your suggestions and have made the revisions and clarifications to address the concerns raised. We would be more than willing to address any additional concerns you may have.
Response to Weakness1 & Weakness 2
Thank you immensely for your time and efforts, as well as the helpful and constructive feedback! Here, we give point-by-point responses to your comments.
Weakness 1& Question1: CFE as a prompt-level reward-shaping heuristicThank you for highlighting this distinction. CFE should be understood as a prompt-level reward-shaping heuristic, rather than an estimate of the value of an individual feedback request. Equation (2) compares calling and skipping rollouts within the same prompt, holding the question fixed. However, the two groups may reach different search histories before choosing whether to request feedback. The resulting success gap therefore reflects both feedback effects and differences in the states in which feedback is requested. Equation (3) assigns this same prompt-level gap to all calling rollouts, without attributing value to any particular request.
To examine this pre-feedback imbalance, we measured whether the evidence retrieved by comparable decision points contained the gold answer. The gold answer was present in 12% of calling rollouts and 44% of skipping rollouts, a difference of 32 percentage points. This pattern is consistent with the agent requesting feedback more often when the evidence collected so far does not contain the answer. It also shows that the two groups differ before feedback is provided, so their subsequent success gap cannot be attributed solely to feedback.
Accordingly, CFE provides an aggregate signal for shaping feedback-seeking behavior within each prompt; it does not identify the causal benefit of a specific request at a particular state.
Weakness 2: Fixed-prefix validation of the RDPO preference labelsEstimating the effect of feedback itself would require comparing continuations with and without feedback from the same prefix. We ran exactly the prefix-matching experiment described and draw a clear conclusion: the preferred feedback does lead to higher success.
Setup. We sampled 750 RDPO pairs from our 2Wiki run (150 from each of iterations 1–5). For each pair we froze the agent checkpoint and the trajectory up to the first feedback request, injected one feedback, and continued the rollout 8 times with the training sampling settings.
Overall, chosen feedback yielded a recovery rate 16.2 percentage points higher than rejected feedback. Recovery with chosen feedback rose from 63.5% in iteration 1 to 74.7% in iteration 5. The gap peaked at 23.7 percentage points in iteration 4 and was 20.3 percentage points in iteration 5. The larger gaps in later iterations are consistent with RDPO training improving the critic’s ability to provide feedback that helps the agent recover.
We also asked DeepSeek V4.1 Flash to compare the two feedback responses in each of the same 750 RDPO pairs, with the original preference labels hidden.
The judge favored chosen feedback, consistent with the rollout results. The judge assessed the feedback itself, while the rollouts measured the agent’s subsequent performance. Together, the two evaluations provide complementary evidence that the RDPO preference labels identify feedback that is more useful to the agent.