- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents EVOKE, a post-training method that that inject goal-diversity in policy optimization, and using contrastive ranking to train the model. The paper shows not the strongest but promising results under three small open source models and three types of benchmarks. The author also provides ablation to study how goal intervention and diversity affects the model's performance.
Critical strengths
-
The entire methodology is simple yet effective. Between all four stages, they are fairly easy to implement and do not depends on specific hardcoded conditions too much. E.g. Multi-positive ranking within each goal is a simple and intuitive design but it aligns with the entire storyline, and the ablation shows that we need multi-positive ranking.
-
The authors put significant efforts on benchmarking the idea. We can see that this training method does not perform the best for all tasks, but there are a large amount of settings where EVOKE performs the best or seconds best, comparing to different post training variants and world-model methods.
-
Good ablation studies to present the components of EVOKE.
Critical weaknesses
-
It looks like the algorithm is RL related, and it requires goal intervention to update the model, so that it might end up more cost. I think the paper lacks discussion on the cost estimation (dollars & FLOPs), comparing to other baselines.
-
The entire section 3 is about EVOKE, where I did not find out where the exact algorithm of EVOKE is. I think the authors would provide a clearer version of algorithm description. Differences between EVOKE and other RL algorithms can be potentially clearer, in section 2 or section 3.
Questions for authors
-
Any comparison to iterative DPO or something similar? If there are baselines that is an variant of iterative DPO, it's great to point it out. The reason is that the entire looks to me like a variant of iterative DPO, where DPO can use Plackett-Luce model ( a more general version of Bradley Terry model) to deal with preference under multiple choices. Other than that, step2 and step3 serves a ranking of the choices.
-
Follow question 1, it would be great if the author clarifies more on the algorithm part. Give the figure, I saw policy rollout. Are we using any policy gradient algorithm over here (e.g. PPO, GRPO, etc)? Or it's the same pipeline of SFT, but with ranking and a customized objective? If it's latter, than I personally would categorize it as a variant of DPO, where the authors should explain more clearly, either mention the connection between EVOKE and DPO, or correct me over here for clearer context.
3 comments
Response to Reviewer ppgv Q1: Relationship to DPO and Additional Comparison (Part 2/3)
Q1) How does EVOKE relate to iterative DPO and preference models over multiple choices?
We agree that EVOKE is closely related to iterative preference learning, and that Plackett–Luce preference models can handle multiple candidates. EVOKE's contribution centers on fixed-state goal interventions that organize action preferences across goals, together with iterative collection of the current policy's hard negatives.
With standard single-goal supervision, a familiar context can be associated with a routine action. EVOKE instead holds the state, history, and candidate actions fixed and changes the goal. When preferred actions differ, a goal-blind ranking cannot satisfy both contexts. This supervision structure can also be used with standard DPO.
We provide an additional comparison using Qwen2.5-3B on ALFWorld. Both DPO variants use β = 0.1, the same initialization and matched update counts. Evaluation counts successful games directly: 140 Seen and 134 Unseen games.
Table R2. Controlled comparison of goal interventions and preference objectives.
| Training setting | Seen | Unseen |
|---|---|---|
| Original goals + standard DPO | 82.14% (115/140) | 78.36% (105/134) |
| Goal interventions + standard DPO | 90.00% (126/140) | 85.82% (115/134) |
| Goal interventions + EVOKE ranking | 90.00% (126/140) | 91.79% (123/134) |
Adding goal interventions improves standard DPO by 7.46 percentage points on unseen games. With goal-intervention supervision, EVOKE's ranking objective gives a further 5.97-point improvement.
The two DPO arms use the collected two-round data, testing the intervention and objective within our data pipeline. At the objective level, standard DPO uses a frozen-reference log-ratio, whereas EVOKE uses reference-free, length-normalized scores with a positive-set term and a pairwise term. These results clarify both the connection to DPO and the contribution of the fixed-state supervision design.
Response to Reviewer ppgv W1: Compute and Interaction Costs (Part 3/3)
W1) Could goal interventions make the method more expensive than RL or world-model pipelines?
We provide an estimated resource comparison at a common 7B policy scale. EVOKE's data-construction budget includes goal and candidate generation, 72B outcome verification, environment restoration or replay, and short continuations.
Table R1. Estimated resource budgets. Setup includes acquisition and policy training, including initialization; deployment compute is per 1,000 decisions.
| Resource | EVOKE | GRPO | ITP-R |
|---|---|---|---|
| Data acquisition (EFLOPs) | 2.11 | 6.89 | 3.74 |
| Policy training (EFLOPs) | 1.16 | 35.14 | 9.65 |
| Total setup (EFLOPs) | 3.27 | 42.03 | 13.39 |
| Environment interactions | 164.4k | 775.7k | 73.4k |
| Deployment (PFLOPs / 1,000 decisions) | 14.56 | 14.56 | 130.8 |
All methods use the same compute approximation: approximately 2NT FLOPs for forward computation and 6NT for training, with N parameters and T processed tokens.
EVOKE uses approximately 12.9× less setup compute than GRPO and 4.1× less than ITP-R, 4.7× fewer interactions than GRPO, and 9.0× less deployment compute than ITP-R. Its branching expense is incurred while constructing preferences. Training operates on those preference groups, and deployment runs the policy without an annotator or lookahead world model.
This separates the computational effect of goal interventions from the much larger cost of repeated online RL optimization or lookahead inference, and makes the efficiency comparison explicit beyond the number of labeled states.
Response to Reviewer ppgv W2 and Q2: Training Algorithm (Part 1/3)
Thank you for recognizing the simplicity of the framework and the breadth of the experiments. Your questions help us explain the optimization procedure and its relationship to iterative preference learning more directly.
W2 / Q2) What is the exact algorithm, and are rollouts used for PPO or GRPO updates?
EVOKE uses supervised preference updates, not a policy-gradient update. Rollouts collect states visited by the current policy; candidate executions provide grounded outcomes for preference labels. Training then optimizes action rankings conditioned on each goal.
The complete loop is:
For each goal, action scores are mean token log-probabilities. The set term increases the total probability mass of positive actions under the candidate softmax; the pairwise logistic term ranks every positive above every competitor. There is no PPO/GRPO return or advantage estimator, and no frozen reference-policy log-ratio in EVOKE's objective.
Updated policies visit new states and expose new high-probability errors, which supply the next round's preference groups. This iterative data collection is the role of policy rollouts in our framework.