- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Contrastive Privileged Action Distillation (CPAD) for long-horizon language-model agents. It groups on-policy decision occurrences using normalized state keys, estimates action preferences from downstream trajectory outcomes, and selects groups with a supported preferred and inferior action. At selected decisions, a frozen pre-update actor scores the same sampled response under two identically worded prompts that differ in the guiding action. The token-level log-probability contrast is clipped and added to a decision-level outcome advantage. The correction can change its sign; ordinary rollout and deployment prompts contain no privileged guidance. Experiments use Qwen2.5-Instruct at 1.5B, 3B, and 7B on ALFWorld and WebShop. CPAD improves final success and validation AUC over the matched outcome-only baseline in all six seen-task/WebShop settings, with reported interaction reductions of 24.1-48.7% at 150 updates. It is competitive rather than uniformly best against stronger methods. Additional experiments examine positive-only and negative-only guidance, selection and clipping sensitivity, response-mean and shuffled-token corrections, random corrections, and training efficiency. Table 1 reports means and standard deviations from five evaluations of the same checkpoint.
Critical strengths
- The selector and contrast use a consistent unit of evidence. The method identifies a local action comparison in matched decision groups, then uses that comparison both to choose where distillation acts and to construct the two guidance contexts. Scoring the same sampled response under both branches avoids imitating a different trajectory's response, and the identical prompt wording gives a concrete way to reduce shared context effects. The contribution is a useful integration of state-level credit information and token-level privileged supervision. For related-work positioning, the authors could discuss Ning Yang et al., Token-Importance Guided Direct Preference Optimization (ICLR 2026; https://openreview.net/forum?id=cMEnMVvMw9). Its semantic token-importance weighting is a relevant conceptual comparison for dense token-level supervision; CPAD instead derives token corrections from paired action guidance at selected decisions. Clarifying these differences would help position the contribution.
- The experimental controls go beyond an endpoint comparison with GRPO. In the WebShop 1.5B token ablation, replacing CPAD corrections by response means or shuffling them reduces AUC from 48.90 to 39.46 or 39.43 while retaining the selector and paired prompts. The random-correction control also fails to recover the full gain. Appendix F includes privileged scoring in the training-time comparisons and reports higher AUC than HGPO, GiGPO, and GraphGPO within common time budgets across all six settings. These controls support the usefulness of informative, token-aligned corrections and the practical value of the overall method, even though their reliability across training seeds remains untested.
Critical weaknesses
- The uncertainty estimates do not measure training reproducibility. Appendix H specifies data/environment/rollout seeds of 0/0/0 and states that Table 1's error bars come from five evaluations of the same checkpoint. This covers sampling noise at evaluation, not variability in the learned policy or selected action pairs. Narrow comparisons should therefore be interpreted cautiously: for example, CPAD versus GraphGPO on ALFWorld unseen at 3B is 90.1 versus 89.9, and CPAD+ slightly exceeds paired CPAD on WebShop 3B (73.4 versus 73.3). The paper correctly presents the method as competitive, but needs independent training seeds before ranking and efficiency conclusions can be treated as robust.
- Decision comparability and the benefit of pairing need stronger validation. Default matching uses h=0, and an action may be supported by only one trajectory. Terminal success can reflect the continuation policy or information in the full history that is absent from the matching key, so the observed action-success association need not be a reliable local preference. Appendix D explores history and support thresholds, but does not directly measure preference accuracy or specify the benchmark-specific contents of the normalized state representation in enough detail. Also, the paired signal adds two components at unit weight, whereas the one-sided controls retain one; Appendix I compares equal preference scale through half the raw contrast. Scale-matched one-sided controls and a random or coverage-matched decision selector would better isolate cancellation from signal magnitude and selective intervention. Proposition I.2 requires the contrast to preserve continuation-value ordering and uses an idealized KL update, so it does not guarantee improvement for the implemented clipped, shared-parameter actor.
Questions for authors
- Can you report at least a representative comparison over independent training seeds against GraphGPO and the matched outcome-only baseline, including success AUC, final success, and total training cost? Please distinguish training variance from the five repeated evaluations in Table 1.
- What exactly enters the normalized decision-state key for ALFWorld and WebShop? Does it retain inventory, progress, product options, prior actions, and all information needed to compare admissible actions? Please provide matched examples and an audit of whether the selected preferred action remains preferable under comparable continuations.
- Do the gains persist against one-sided guidance with matched correction RMS or a matched lambda sweep, and against random or coverage-matched decision selection? These comparisons would distinguish paired cancellation and evidence-based selection from simply changing correction strength.
- How frequently does the additive correction reverse the outcome-advantage sign, and are those reversals concentrated on decisions whose local value differs from the terminal outcome? Please connect this diagnostic to the alignment condition in Proposition I.2 and make the conditional scope of the theory explicit in the main text.
6 comments
To Q2: Matching fields and local-preference audit
For the normalized decision-state key. Section 3.1 specifies the matching fields. Within each task's rollout batch, decisions with identical keys form a group:
| Environment | Matching-key fields |
|---|---|
| ALFWorld | Task identity, normalized current observation, normalized and sorted admissible-action set, and configured observation–action history |
| WebShop | The same fields, plus selected product options from the environment session |
Here, h is the number of recent observation–action pairs included in the key. The default h=0 uses the current-state fields, while the actor retains up to two previous observation–action pairs in its ordinary context. Inventory and progress enter when represented in the observation; WebShop additionally records selected options explicitly. Complete hidden state and remaining budget are not separate key fields, so decisions differing in information outside these fields can share a key. We therefore test the resulting preferences directly under controlled continuations.
For the comparable-continuation audit. Training-time matching uses equality of the key above. For each audited action pair in Appendix J, Table 10, we instead restore the same complete WebShop state, policy context, and remaining budget before executing either action. We freeze the WebShop–1.5B CPAD seed-1 policy at update 150 and sample eight fresh continuations per action, separately from the discovery rollouts that supplied the pair.
In the table, minimum support counts distinct discovery trajectories supporting each action; “Pairs / tasks” counts audited source-state/action-pair units and represented tasks. Success gaps are preferred minus inferior continuation success in percentage points (pp), averaged within tasks and then equally across tasks. The 95% confidence intervals resample tasks. History/support strata can overlap; a single-task stratum has no task-bootstrap interval.
| History h | Min. support per action | Pairs / tasks | Success gap, pp [95% CI] |
|---|---|---|---|
| 0 | 1 | 13 / 9 | +56.7 [28.5, 84.0] |
| 0 | 2 | 5 / 5 | +30.0 [−5.0, 72.5] |
| 1 | 1 | 12 / 8 | +72.7 [37.5, 97.7] |
| 1 | 2 | 2 / 2 | +43.8 [−12.5, 100.0] |
| 2 | 1 | 8 / 6 | +66.0 [31.3, 100.0] |
| 2 | 2 | 1 / 1 | +12.5 [—] |
| All audited units | — | 22 / 9 | +58.1 [30.3, 84.3] |
Under the default rule, fresh-continuation success is higher for the preferred action in 8 of 13 pairs, tied in 5, and lower in none. Its task-averaged advantage is 56.7 percentage points; the pooled audit also gives a positive advantage of 58.1 points. These measurements directly support the local value of the selected preferences.
For matched examples. The following cases show what the matching fields preserve and what the resulting local guidance contributes.
Selecting the missing product option. The task requests lotion with a specified scent and size. On the current product page, the scent is selected but the required size is still missing. Matched rollouts supply two actions: select that size, or buy immediately. From the same restored state with 12 steps remaining, selecting the size yields 8/8 successful continuations, versus 0/8 for buying immediately. Selecting the size leaves the observation text and admissible-action set unchanged, but updates the selected-options field. That field therefore distinguishes configuration progress that the page text alone misses.
Improving a successful trajectory. A discovery rollout searching for a suitable jumpsuit chose product A from a search-results page and eventually succeeded. At the same page, default matching identifies product B as the preferred alternative. Replaying the two choices from the same complete state with nine steps remaining gives 8/8 successful continuations for B and 0/8 for A. This shows how CPAD identifies a local improvement within a trajectory already rewarded as successful.
To Q3: Correction scale and decision selection
For one-sided guidance with matched correction scale. CPAD+ retains the preferred-view component, and CPAD− retains the inferior-control component. We match the root mean square (RMS) of each control's clipped correction to the clipped paired reference computed on that control's current batch, over selected valid response tokens. This preserves relative token weights while controlling correction magnitude; both controls achieve numerical-precision RMS matching.
The comparisons in Section 4.4 and Appendix K, Table 13 use WebShop–1.5B, seed 1, the same selection rules and outcome anchor, and 150 updates. Success nAUC covers updates 0–150; final success is evaluated at update 150:
| Method | Success nAUC | Final success |
|---|---|---|
| RMS-matched CPAD+ | 29.71 | 62.2% |
| RMS-matched CPAD− | 4.66 | 7.6% |
| CPAD | 47.09 | 66.4% |
After RMS matching, CPAD retains nAUC advantages of 17.39 and 42.44 points over the positive-only and negative-only controls, together with final-success gains of 4.2 and 58.8 percentage points. The benefit of pairing therefore persists when correction magnitude is controlled.
For random or coverage-matched decision selection. CPAD's action evidence defines both the guidance location and its content: a correction is defined where matched rollouts supply a supported preferred–inferior action pair, and every such decision is used. Randomly drawing the same number of decisions from this eligible set returns the entire set; positions outside it have no action pair under the original construction. A random-location control with a different selected set cannot preserve both CPAD's action-pair construction and its selection coverage.
This coupling motivates the matching-history and support ablations. Varying h changes which decisions share a group, which action comparisons have sufficient support, and consequently which decisions receive guidance. Raising the support threshold similarly changes the evidence required for an action pair. These ablations directly test how matching information and supervision availability affect learning while retaining the paired-guidance mechanism.
The paper's original seed-0 WebShop–1.5B matching study in Appendix D, Table 3 reports:
| Setting | Mean decision coverage | Success nAUC | Final success |
|---|---|---|---|
| GRPO | — | 31.54 | 62.8% |
| Default, h=0 | 14.94% | 48.90 | 68.6% |
| h=1 | 9.26% | 46.47 | 65.0% |
| h=2 | 7.22% | 44.65 | 72.6% |
| Minimum support 2 per action | 4.69% | 47.17 | 71.4% |
All four selection settings exceed GRPO in both nAUC and final success, including the setting with only 4.69% decision coverage. The audit in Q2 independently tests the preferences produced by these matching rules, connecting the coverage–performance results to local continuation value.
Separate controls keep the selection rule fixed and change the correction content: response averaging, token shuffling, and random correction values yield nAUCs of 39.46, 39.43, and 36.28, versus 48.90 for CPAD. These results further support the value of informative, token-aligned guidance at the selected decisions.
To Q1: Independent training seeds and cost
For performance across independent training seeds. Across WebShop–1.5B and 3B, CPAD achieves higher success nAUC than both GRPO and GraphGPO at each of two independent training seeds. Appendix L, Table 14 compares all three methods at seeds 0 and 1 for each model size, using the same 500-goal evaluation pool and 150-update budget. Success nAUC is the trapezoidal integral over updates 0–150 divided by 150 and expressed on a 0–100 scale; final success is the update-150 evaluation.
Success nAUC
Final success (%)
CPAD's nAUC advantage over GraphGPO is reproduced at both seeds and both model sizes: +2.83 / +2.89 points at 1.5B and +8.52 / +15.93 at 3B. At 3B, its final-success advantage over GraphGPO also recurs: +0.6 and +3.0 percentage points. Against GRPO, CPAD achieves higher final success at both seeds in both model sizes.
For training versus evaluation variance. The means and sample standard deviations above are computed separately for each model size across the two independently trained policies. Table 1's five evaluations instead measure evaluation variability at one fixed trained checkpoint.
For training cost and learning efficiency. The independent WebShop–1.5B seed-1 runs use 36.61 GPU-hours for CPAD, 37.11 for GraphGPO, and 42.23 for GRPO, including evaluation, checkpoint saving, and repeated computation within the recorded training loop (Appendix L, Table 15).
The manuscript's WebShop–7B comparison further shows how the learning advantage translates into lower cost and stronger performance within a fixed training-time budget (Appendix F; Appendix L, Table 16). Training time below includes privileged rescoring and excludes evaluation and checkpoint saving. Reaching the 70% success target requires two consecutive evaluations at or above 70%, with cost measured at the second evaluation:
CPAD uses 19.9% less training time for 150 updates and reaches the 70% success target with 64.9% less training time than GraphGPO. Within the common budget, it also improves success nAUC by 18.89 points. The common-time nAUC advantage over GraphGPO extends across all six ALFWorld/WebShop configurations (Appendix F, Table 5).