acceptodds
Under review as a conference paper at ICLR 2027

Sparse Where, Dense Within: Contrastive Privileged Action Distillation for Long-Horizon Agents

Abstract

Training long-horizon language model agents from terminal rewards is challenging because task outcomes provide limited guidance for individual decisions. On-policy distillation can supply dense token-level supervision using privileged training information. However, dense feedback alone does not identify which decisions warrant additional guidance, and positive guidance can conflate action preferences with the effects of added context. We introduce Contrastive Privileged Action Distillation (CPAD), which uses action comparisons supported by rollout outcomes to determine both where and how to distill. CPAD matches repeated decision states across on-policy trajectories and selects only those where downstream outcomes support a preferred action over an inferior alternative. At each selected decision, the same model re-scores its sampled response under two contexts that differ only in the guiding action. Their token-level log-probability difference augments the outcome-based advantage, making supervision selective across decisions and dense within each selected response. Privileged guidance is used only during training. Experiments on ALFWorld and WebShop with Qwen2.5 models at 1.5B, 3B, and 7B scales show competitive final performance against structured credit-assignment methods. Compared with a matched outcome-based RL baseline, CPAD improves final success and the area under the 150-update validation success curve in all six settings on ALFWorld seen tasks and WebShop. At 150 updates, final success increases by up to 24.4 and 11.4 percentage points on the two benchmarks, respectively, with 24-49% fewer training interactions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
large language model agents, reinforcement learning, on-policy distillation, privileged information, long-horizon decision making

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SgDGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Contrastive Privileged Action Distillation (CPAD) for long-horizon language-model agents. It groups on-policy decision occurrences using normalized state keys, estimates action preferences from downstream trajectory outcomes, and selects groups with a supported preferred and inferior action. At selected decisions, a frozen pre-update actor scores the same sampled response under two identically worded prompts that differ in the guiding action. The token-level log-probability contrast is clipped and added to a decision-level outcome advantage. The correction can change its sign; ordinary rollout and deployment prompts contain no privileged guidance. Experiments use Qwen2.5-Instruct at 1.5B, 3B, and 7B on ALFWorld and WebShop. CPAD improves final success and validation AUC over the matched outcome-only baseline in all six seen-task/WebShop settings, with reported interaction reductions of 24.1-48.7% at 150 updates. It is competitive rather than uniformly best against stronger methods. Additional experiments examine positive-only and negative-only guidance, selection and clipping sensitivity, response-mean and shuffled-token corrections, random corrections, and training efficiency. Table 1 reports means and standard deviations from five evaluations of the same checkpoint.

Critical strengths
  1. The selector and contrast use a consistent unit of evidence. The method identifies a local action comparison in matched decision groups, then uses that comparison both to choose where distillation acts and to construct the two guidance contexts. Scoring the same sampled response under both branches avoids imitating a different trajectory's response, and the identical prompt wording gives a concrete way to reduce shared context effects. The contribution is a useful integration of state-level credit information and token-level privileged supervision. For related-work positioning, the authors could discuss Ning Yang et al., Token-Importance Guided Direct Preference Optimization (ICLR 2026; https://openreview.net/forum?id=cMEnMVvMw9). Its semantic token-importance weighting is a relevant conceptual comparison for dense token-level supervision; CPAD instead derives token corrections from paired action guidance at selected decisions. Clarifying these differences would help position the contribution.
  2. The experimental controls go beyond an endpoint comparison with GRPO. In the WebShop 1.5B token ablation, replacing CPAD corrections by response means or shuffling them reduces AUC from 48.90 to 39.46 or 39.43 while retaining the selector and paired prompts. The random-correction control also fails to recover the full gain. Appendix F includes privileged scoring in the training-time comparisons and reports higher AUC than HGPO, GiGPO, and GraphGPO within common time budgets across all six settings. These controls support the usefulness of informative, token-aligned corrections and the practical value of the overall method, even though their reliability across training seeds remains untested.
Critical weaknesses
  1. The uncertainty estimates do not measure training reproducibility. Appendix H specifies data/environment/rollout seeds of 0/0/0 and states that Table 1's error bars come from five evaluations of the same checkpoint. This covers sampling noise at evaluation, not variability in the learned policy or selected action pairs. Narrow comparisons should therefore be interpreted cautiously: for example, CPAD versus GraphGPO on ALFWorld unseen at 3B is 90.1 versus 89.9, and CPAD+ slightly exceeds paired CPAD on WebShop 3B (73.4 versus 73.3). The paper correctly presents the method as competitive, but needs independent training seeds before ranking and efficiency conclusions can be treated as robust.
  2. Decision comparability and the benefit of pairing need stronger validation. Default matching uses h=0, and an action may be supported by only one trajectory. Terminal success can reflect the continuation policy or information in the full history that is absent from the matching key, so the observed action-success association need not be a reliable local preference. Appendix D explores history and support thresholds, but does not directly measure preference accuracy or specify the benchmark-specific contents of the normalized state representation in enough detail. Also, the paired signal adds two components at unit weight, whereas the one-sided controls retain one; Appendix I compares equal preference scale through half the raw contrast. Scale-matched one-sided controls and a random or coverage-matched decision selector would better isolate cancellation from signal magnitude and selective intervention. Proposition I.2 requires the contrast to preserve continuation-value ordering and uses an idealized KL update, so it does not guarantee improvement for the implemented clipped, shared-parameter actor.
Questions for authors
  1. Can you report at least a representative comparison over independent training seeds against GraphGPO and the matched outcome-only baseline, including success AUC, final success, and total training cost? Please distinguish training variance from the five repeated evaluations in Table 1.
  2. What exactly enters the normalized decision-state key for ALFWorld and WebShop? Does it retain inventory, progress, product options, prior actions, and all information needed to compare admissible actions? Please provide matched examples and an audit of whether the selected preferred action remains preferable under comparable continuations.
  3. Do the gains persist against one-sided guidance with matched correction RMS or a matched lambda sweep, and against random or coverage-matched decision selection? These comparisons would distinguish paired cancellation and evidence-based selection from simply changing correction strength.
  4. How frequently does the additive correction reverse the outcome-advantage sign, and are those reversals concentrated on decisions whose local value differs from the terminal outcome? Please connect this diagnostic to the alignment condition in Proposition I.2 and make the conditional scope of the theory explicit in the main text.
6 comments
AuthorsOpenReview ↗

To Q1: Independent training seeds and cost

For performance across independent training seeds. Across WebShop–1.5B and 3B, CPAD achieves higher success nAUC than both GRPO and GraphGPO at each of two independent training seeds. Appendix L, Table 14 compares all three methods at seeds 0 and 1 for each model size, using the same 500-goal evaluation pool and 150-update budget. Success nAUC is the trapezoidal integral over updates 0–150 divided by 150 and expressed on a 0–100 scale; final success is the update-150 evaluation.

Success nAUC

ModelMethodSeed 0Seed 1Mean ± SD
1.5BGRPO31.5435.0733.31 ± 2.50
1.5BGraphGPO46.0744.2145.14 ± 1.32
1.5BCPAD48.9047.0948.00 ± 1.28
3BGRPO33.9039.1736.53 ± 3.73
3BGraphGPO42.6040.0441.32 ± 1.81
3BCPAD51.1255.9753.55 ± 3.43

Final success (%)

ModelMethodSeed 0Seed 1Mean ± SD
1.5BGRPO62.855.859.3 ± 4.95
1.5BGraphGPO74.071.072.5 ± 2.12
1.5BCPAD68.666.467.5 ± 1.56
3BGRPO63.667.865.7 ± 2.97
3BGraphGPO71.673.872.7 ± 1.56
3BCPAD72.276.874.5 ± 3.25

CPAD's nAUC advantage over GraphGPO is reproduced at both seeds and both model sizes: +2.83 / +2.89 points at 1.5B and +8.52 / +15.93 at 3B. At 3B, its final-success advantage over GraphGPO also recurs: +0.6 and +3.0 percentage points. Against GRPO, CPAD achieves higher final success at both seeds in both model sizes.

For training versus evaluation variance. The means and sample standard deviations above are computed separately for each model size across the two independently trained policies. Table 1's five evaluations instead measure evaluation variability at one fixed trained checkpoint.

For training cost and learning efficiency. The independent WebShop–1.5B seed-1 runs use 36.61 GPU-hours for CPAD, 37.11 for GraphGPO, and 42.23 for GRPO, including evaluation, checkpoint saving, and repeated computation within the recorded training loop (Appendix L, Table 15).

The manuscript's WebShop–7B comparison further shows how the learning advantage translates into lower cost and stronger performance within a fixed training-time budget (Appendix F; Appendix L, Table 16). Training time below includes privileged rescoring and excludes evaluation and checkpoint saving. Reaching the 70% success target requires two consecutive evaluations at or above 70%, with cost measured at the second evaluation:

WebShop–7B metricGraphGPOCPAD
Training time for 150 updates17.27 h13.84 h
Training time to the 70% success target12.80 h4.49 h
Success nAUC, common 13.84-hour training budget45.3964.28

CPAD uses 19.9% less training time for 150 updates and reaches the 70% success target with 64.9% less training time than GraphGPO. Within the common budget, it also improves success nAUC by 18.89 points. The common-time nAUC advantage over GraphGPO extends across all six ALFWorld/WebShop configurations (Appendix F, Table 5).

AuthorsOpenReview ↗

To Q2: Matching fields and local-preference audit

For the normalized decision-state key. Section 3.1 specifies the matching fields. Within each task's rollout batch, decisions with identical keys form a group:

EnvironmentMatching-key fields
ALFWorldTask identity, normalized current observation, normalized and sorted admissible-action set, and configured observation–action history
WebShopThe same fields, plus selected product options from the environment session

Here, h is the number of recent observation–action pairs included in the key. The default h=0 uses the current-state fields, while the actor retains up to two previous observation–action pairs in its ordinary context. Inventory and progress enter when represented in the observation; WebShop additionally records selected options explicitly. Complete hidden state and remaining budget are not separate key fields, so decisions differing in information outside these fields can share a key. We therefore test the resulting preferences directly under controlled continuations.

For the comparable-continuation audit. Training-time matching uses equality of the key above. For each audited action pair in Appendix J, Table 10, we instead restore the same complete WebShop state, policy context, and remaining budget before executing either action. We freeze the WebShop–1.5B CPAD seed-1 policy at update 150 and sample eight fresh continuations per action, separately from the discovery rollouts that supplied the pair.

In the table, minimum support counts distinct discovery trajectories supporting each action; “Pairs / tasks” counts audited source-state/action-pair units and represented tasks. Success gaps are preferred minus inferior continuation success in percentage points (pp), averaged within tasks and then equally across tasks. The 95% confidence intervals resample tasks. History/support strata can overlap; a single-task stratum has no task-bootstrap interval.

History hMin. support per actionPairs / tasksSuccess gap, pp [95% CI]
0113 / 9+56.7 [28.5, 84.0]
025 / 5+30.0 [−5.0, 72.5]
1112 / 8+72.7 [37.5, 97.7]
122 / 2+43.8 [−12.5, 100.0]
218 / 6+66.0 [31.3, 100.0]
221 / 1+12.5 [—]
All audited units—22 / 9+58.1 [30.3, 84.3]

Under the default rule, fresh-continuation success is higher for the preferred action in 8 of 13 pairs, tied in 5, and lower in none. Its task-averaged advantage is 56.7 percentage points; the pooled audit also gives a positive advantage of 58.1 points. These measurements directly support the local value of the selected preferences.

For matched examples. The following cases show what the matching fields preserve and what the resulting local guidance contributes.

Selecting the missing product option. The task requests lotion with a specified scent and size. On the current product page, the scent is selected but the required size is still missing. Matched rollouts supply two actions: select that size, or buy immediately. From the same restored state with 12 steps remaining, selecting the size yields 8/8 successful continuations, versus 0/8 for buying immediately. Selecting the size leaves the observation text and admissible-action set unchanged, but updates the selected-options field. That field therefore distinguishes configuration progress that the page text alone misses.

Improving a successful trajectory. A discovery rollout searching for a suitable jumpsuit chose product A from a search-results page and eventually succeeded. At the same page, default matching identifies product B as the preferred alternative. Replaying the two choices from the same complete state with nine steps remaining gives 8/8 successful continuations for B and 0/8 for A. This shows how CPAD identifies a local improvement within a trajectory already rewarded as successful.

AuthorsOpenReview ↗

To Q3: Correction scale and decision selection

For one-sided guidance with matched correction scale. CPAD+ retains the preferred-view component, and CPAD− retains the inferior-control component. We match the root mean square (RMS) of each control's clipped correction to the clipped paired reference computed on that control's current batch, over selected valid response tokens. This preserves relative token weights while controlling correction magnitude; both controls achieve numerical-precision RMS matching.

The comparisons in Section 4.4 and Appendix K, Table 13 use WebShop–1.5B, seed 1, the same selection rules and outcome anchor, and 150 updates. Success nAUC covers updates 0–150; final success is evaluated at update 150:

MethodSuccess nAUCFinal success
RMS-matched CPAD+29.7162.2%
RMS-matched CPAD−4.667.6%
CPAD47.0966.4%

After RMS matching, CPAD retains nAUC advantages of 17.39 and 42.44 points over the positive-only and negative-only controls, together with final-success gains of 4.2 and 58.8 percentage points. The benefit of pairing therefore persists when correction magnitude is controlled.

For random or coverage-matched decision selection. CPAD's action evidence defines both the guidance location and its content: a correction is defined where matched rollouts supply a supported preferred–inferior action pair, and every such decision is used. Randomly drawing the same number of decisions from this eligible set returns the entire set; positions outside it have no action pair under the original construction. A random-location control with a different selected set cannot preserve both CPAD's action-pair construction and its selection coverage.

This coupling motivates the matching-history and support ablations. Varying h changes which decisions share a group, which action comparisons have sufficient support, and consequently which decisions receive guidance. Raising the support threshold similarly changes the evidence required for an action pair. These ablations directly test how matching information and supervision availability affect learning while retaining the paired-guidance mechanism.

The paper's original seed-0 WebShop–1.5B matching study in Appendix D, Table 3 reports:

SettingMean decision coverageSuccess nAUCFinal success
GRPO—31.5462.8%
Default, h=014.94%48.9068.6%
h=19.26%46.4765.0%
h=27.22%44.6572.6%
Minimum support 2 per action4.69%47.1771.4%

All four selection settings exceed GRPO in both nAUC and final success, including the setting with only 4.69% decision coverage. The audit in Q2 independently tests the preferences produced by these matching rules, connecting the coverage–performance results to local continuation value.

Separate controls keep the selection rule fixed and change the correction content: response averaging, token shuffling, and random correction values yield nAUCs of 39.46, 39.43, and 36.28, versus 48.90 for CPAD. These results further support the value of informative, token-aligned guidance at the selected decisions.

Reviewer_H9jLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes CPAD to improve local credit assignment for long-horizon language model agents. It matches decision states across on-policy rollouts and selects preferred/inferior actions using trajectory outcomes and continuation quality. The frozen current actor scores the original response under the two action-guidance contexts, and their token-level log-probability difference augments the outcome advantage. Experiments cover ALFWorld, WebShop, and three Qwen2.5 model sizes. Table 1 shows substantial gains over matched GRPO, but advantages over GraphGPO are inconsistent. The appendices provide single-branch, token-correction, and efficiency analyses. The central unresolved issue is whether the action preferences and token contrasts reliably align with local continuation value.

Critical strengths
  1. The method addresses a real credit-assignment problem without requiring a stronger teacher. Action pairs come from current rollouts and are scored by the same actor. In Table 1, at 3B, ALFWorld seen success increases from GRPO’s 73.9% to 95.6%, and WebShop success from 61.9% to 73.3%, supporting practical training value.

  2. The ablations and efficiency accounting are substantive. In Table 4, CPAD achieves 48.90% AUC, versus 39.46% for response averaging and 39.43% for token shuffling, supporting the role of token alignment. Appendix F includes privileged scoring in training time, making the efficiency comparison more credible than update counts alone.

Critical weaknesses
  1. Local action preferences are uncalibrated. Default matching excludes history and permits one supporting trajectory per action (Appendix H). Terminal outcomes can mislabel local actions, yet preference accuracy is not measured, and the correction can reverse the outcome advantage (§3.3).

  2. Mechanism and robustness evidence is incomplete. Value alignment is assumed rather than tested (Appendix I). Ablations lack scale-matched single-branch and direct action-level controls. Table 1 reports evaluation variance from one checkpoint, leaving small gains over GraphGPO uncertain.

Questions for authors
  1. Which fields constitute the matching state in each environment? Does matching use simulator state beyond the agent’s observations? Can a group merge decisions with different task progress, hidden states, or remaining step budgets?
  2. Can you execute the preferred and inferior actions from identical environment states, then run multiple continuations under the same policy, and report preference-ranking accuracy by action support count and matching-history length?
  3. With the same selector, how do direct empirical action advantages or preferred-action distillation perform? Does CPAD retain its advantage after independently tuning the single branches and matching correction scale?
  4. Was each configuration trained only once? Were parameters in Tables 3 and 4 selected using the main validation pool? Do the gains over GraphGPO persist across training seeds and tasks excluded from tuning?
8 comments
AuthorsOpenReview ↗

To Q1: Matching fields and decision comparability

For the matching fields. Within each task's rollout batch, decisions with identical matching keys form a group. Section 3.1 specifies the fields:

EnvironmentMatching-key fields
ALFWorldTask identity, normalized current observation, normalized and sorted admissible-action set, and configured observation–action history
WebShopThe same fields, plus the selected product options

Here, h denotes the number of recent observation–action pairs included in the matching key. The default h=0 uses the current-state fields, while the actor still receives up to two previous observation–action pairs in its ordinary context.

For information beyond the observation. WebShop reads the selected product options from the environment session. This field records configuration progress that the observation text can miss: in an audited lotion-shopping trajectory, selecting the requested size leaves the observation text and admissible-action set unchanged, while the key's options change from scent=butter to scent=butter, size=8.5 fl oz (pack of 1). In ALFWorld, the admissible-action list is supplied in the actor's ordinary input.

For decision comparability. The key represents decision context through the current observation, admissible actions, configured history, and explicit product options. Decisions differing in unencoded hidden state, progress, or remaining budget can share a key. Q2 directly tests whether the resulting preferences predict better continuation outcomes by restoring the same complete environment state, policy context, and remaining budget before either action.

AuthorsOpenReview ↗

To Q2: Identical-state continuation audit

For comparisons from identical environment states. In the audit (Appendix J, Table 10), each preferred–inferior action pair is tested from one identical complete environment snapshot, with the same pre-action policy context and remaining interaction budget restored before executing either action.

We froze the WebShop–1.5B CPAD seed-1 policy at update 150 and sampled eight fresh continuations per action under that policy. These continuations are separate from the discovery rollouts used to identify the pair, directly measuring which action leads to higher success under controlled continuation conditions.

For preference ranking by action support and matching history. In the table, h is the number of previous observation–action pairs in the matching key. Minimum support counts distinct discovery trajectories supporting each action. An audited action-pair unit consists of a source state and a preferred–inferior action pair; “Pairs / tasks” reports the number of such units and the number of tasks represented.

“Higher / tied / lower” compares the two actions' fresh-continuation success estimates. The success gap is preferred minus inferior success, in percentage points (pp), first averaged within each task and then equally across tasks. The 95% confidence intervals resample tasks. History/support strata can share audited pairs; the last row aggregates all 22 unique units. A single-task stratum has no task-bootstrap interval.

Matching history hMin. support per actionPairs / tasksHigher / tied / lowerSuccess gap, pp [95% CI]
0113 / 98 / 5 / 056.71 [28.47, 84.03]
025 / 52 / 2 / 130.00 [−5.00, 72.50]
1112 / 89 / 2 / 172.66 [37.50, 97.66]
122 / 21 / 0 / 143.75 [−12.50, 100.00]
218 / 65 / 1 / 265.97 [31.25, 100.00]
221 / 11 / 0 / 012.50 [—]
All—22 / 913 / 7 / 258.10 [30.32, 84.26]

The default h=0, minimum-support-1 rule yields higher success in 8 pairs, tied success in 5, and lower success in none, with a task-averaged advantage of 56.71 percentage points. With minimum support 1, all three matching histories also yield positive task-averaged continuation advantages. These results directly support the local value of the selected action preferences under the actual default rule and across the tested matching histories.

A concrete example shows what this local guidance adds to terminal rewards. A successful discovery rollout selected product A while searching for a suitable jumpsuit. Default matching identifies product B as the preferred alternative on the same search-results page: discovery success was 1/1 for B versus 1/5 for A. From the same restored state with nine steps remaining, B yields 8/8 fresh continuation successes and A yields 0/8. CPAD thus identifies a local improvement within a trajectory already rewarded as successful.

AuthorsOpenReview ↗

To Q3: Direct action advantages and correction-scale controls

For direct empirical action advantages with the same selector. This control isolates the value of converting action evidence into token guidance, using the same decision-selection rule and outcome anchor (Section 4.4; Appendix K, Table 12). It replaces privileged token rescoring with the executed action's empirical success rate minus the support-weighted group success rate, broadcast over that response's valid tokens. Its scale is fixed from CPAD training statistics.

Both methods train on WebShop–1.5B with seed 1 for 150 updates. Here, success nAUC is the trapezoidal validation-success integral over updates 0–150, divided by 150 and expressed on a 0–100 scale; final success is the update-150 evaluation:

MethodSuccess nAUCFinal success
Direct empirical action advantage40.9066.6%
CPAD47.0966.4%

CPAD improves nAUC by 6.19 points. Its advantage emerges during learning: success is 48.4% versus 31.4% at update 50 and 56.8% versus 45.6% at update 75. This shows that converting the action evidence into privileged token guidance produces earlier learning gains, complementing the existing response-averaging and token-shuffling controls.

For single-branch tuning and correction-scale matching. We test whether the paired signal's advantage persists at matched effective correction magnitude. CPAD+ retains the preferred-view component, and CPAD− retains the inferior-control component. The controls use batchwise root-mean-square (RMS) matching, without a separate hyperparameter search for each branch. On each control's current batch, we match the RMS of its clipped correction to the clipped paired reference over selected valid response tokens. This preserves relative token weights while controlling magnitude, and both controls achieve numerical-precision RMS matching.

The comparisons in Appendix K, Table 13 use the same WebShop–1.5B seed-1 setup and metrics. Mean reference KL measures divergence from the reference policy over training:

MethodSuccess nAUCFinal successMean reference KL
CPAD+ with matched RMS29.7162.2%0.4828
CPAD− with matched RMS4.667.6%0.2584
CPAD47.0966.4%0.1115

After matching, CPAD exceeds the positive-only and negative-only controls by 17.39 and 42.44 nAUC points, and by 4.2 and 58.8 final-success points, respectively, while maintaining lower mean reference KL. The paired signal's learning and final-performance benefits persist when correction magnitude is controlled.

Reviewer_6ZCiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CPAD for reinforcement learning of agents in LLMs. Existing on-policy distillation methods impose dense token-level supervision on all decisions, failing to distinguish which decisions truly require additional guidance; relying solely on positive supervision also makes it easy for false signals introduced by context to creep in.

CPAD utilizes matched decision states within the same batch rollout to identify decision points where there are superior and inferior actions and where downstream outcomes exhibit clear preferences. For selected decisions, it constructs superior and degraded action as privileged prompts, re-score the same response using a frozen model, and use the difference in token-level log-likelihoods between the two as a local correction term, which is superimposed onto the baseline advantage derived from trajectory rewards. Privileged prompts are used only during training. The original interaction context is preserved during inference and deployment. They conducted experiments on ALFWorld and WebShop using Qwen2.5-1.5B/3B/7B.

Critical strengths
  1. With the same number of update steps, the model achieves a higher AUC than several state-of-the-art credit allocation baselines while using fewer interaction samples and within the same time budget.
  2. The paper identifies two major pain points in existing on-policy distillation: the indiscriminate application of dense supervision and the tendency for unidirectional privileged signals to introduce contextual noise. It proposes using matching decision states combined with positive-negative action comparisons to select supervision locations and construct signals.
Critical weaknesses
  1. The choice of decision-making keys has a significant impact on algorithm performance. Please systematically analyze how different key designs affect grouping quality and performance, and provide practical recommendations for selection.
  2. In environments characterized by a wide variety of observation heights and few successful trajectories, the proportion of selected decisions decreases, and supervisory signals become scarce. However, the lack of quantitative simulations or supplementary experiments for this failure scenario raises doubts about the method’s generalizability.
Questions for authors
  1. As the proportion of selection decisions increases, how does the computational overhead resulting from the two re-scoring passes change?
  2. Can the algorithm be validated on a wider range of agent tasks (such as tool use), demonstrating that it is not limited to ALFWorld and WebShop?
5 comments
AuthorsOpenReview ↗

To Q1: Rescoring cost as selection coverage increases

Both privileged rescoring passes operate on selected responses, so their cost follows the selected token workload. The measurements in Appendix F, Table 7 group all 150 updates of the WebShop–1.5B training run by their observed decision coverage:

Selected-decision coverageUpdatesMean selected response tokensCombined time for both passes
<5%371,8710.50 s
5–15%547,5572.19 s
15–30%3722,2916.38 s
≥30%2242,49412.33 s

Across the six default configurations, the two rescoring passes together account for 1.89–6.49% of training time, excluding evaluation and checkpoint saving. With rescoring included, CPAD achieves higher success AUC than HGPO, GiGPO, and GraphGPO under a common training-time budget in all six configurations (Section 4.5).

AuthorsOpenReview ↗

To Q2: Tool interaction and visual spatial planning

WebShop evaluates web-based tool interaction: the agent uses search[query] and click[item] to search for products, navigate pages, select product options, and complete a purchase.

We extend the evaluation to visual Sokoban with Qwen2.5-VL-3B, adding visual input and spatial planning (Appendix L, Table 18). Both methods use a 6×6 board with one box, a 15-step horizon, 150 training updates, and 256 evaluation tasks under the same task protocol:

MethodSuccess nAUC, updates 0–150Final success
GRPO17.8625.00%
CPAD33.4257.03%

CPAD improves success nAUC by 15.56 points and final success by 32.03 percentage points. These gains demonstrate improved learning over the training budget and higher final task success in visual spatial planning, extending the evidence beyond web tool interaction to a different task and input modality.

AuthorsOpenReview ↗

To W1: Matching keys, preference quality, and practical choices

As specified in Section 3.1, CPAD first groups decision occurrences from the same task and rollout batch by a matching key containing the normalized current observation and normalized, sorted admissible-action set. WebShop additionally includes the selected product options from the environment session. Decisions with identical keys share a group. Within each group, CPAD compares distinct executed actions using their supporting trajectories' success rates and, when needed, continuation quality, retaining groups with a supported preferred–inferior action pair.

The key can also include the most recent h observation–action pairs. Thus, h=0/1/2 controls how much preceding history distinguishes groups; the actor retains up to two previous observation–action pairs in every setting. Adding history makes matching more specific and can reduce the decisions available for preference supervision. We therefore vary h to measure its effect on coverage, learning performance, and the quality of the selected preferences.

The WebShop–1.5B comparison in Appendix D, Table 3 keeps the minimum support at one distinct trajectory per action. Coverage is averaged over updates 1–150, success nAUC covers updates 0–150, and final success is evaluated at update 150:

SettingDecision coverageSuccess nAUCFinal success
Default, h=014.94%48.9068.6%
h=19.26%46.4765.0%
h=27.22%44.6572.6%
GRPO—31.5462.8%

All three matching histories exceed GRPO in both nAUC and final success. Increasing h reduces coverage, while h=0 gives the highest nAUC and h=2 gives the highest final success in this comparison.

To directly assess the resulting preferences, we apply the three history settings to a common discovery pool and audit their selected action pairs under one frozen policy (Appendix J, Table 10). For each pair, we restore the same complete environment state, policy context, and remaining budget, execute either action, and sample eight fresh continuations per action. The table reports preferred-minus-inferior success gaps, averaged within each task and then equally across tasks; 95% bootstrap intervals resample tasks. The history strata can share audited pairs.

Matching history, support ≥1 per actionAction pairs / tasksTask-averaged success gap, pp [95% CI]
h=013 / 9+56.71 [28.47, 84.03]
h=112 / 8+72.66 [37.50, 97.66]
h=28 / 6+65.97 [31.25, 100.00]

The task-averaged preference advantage is positive at all three matching histories. Under the default h=0 rule, the preferred action has higher observed continuation success in 8 pairs, equal success in 5, and lower success in none.

We recommend h=0 with at least one supporting trajectory per action as the default starting point for its higher coverage and nAUC. History can be added when preceding interactions are needed to distinguish decisions, with coverage checked as matching becomes more specific. The training comparisons and continuation audit support CPAD's effectiveness across the tested matching granularities.

Comments on the paper

1 comment
AuthorsOpenReview ↗

To all reviewers: Local preference quality, paired guidance, and learning efficiency

CPAD converts outcome-supported local action comparisons into token-level guidance: the same action evidence determines where supervision acts and how its preferred–inferior contrast is constructed. The revised manuscript brings together direct preference audits, controlled training comparisons, and broader evaluations to establish the value of this construction. Additions and substantive changes are highlighted in blue.

1. Selected action preferences identify useful local improvements (Section 3.1; Appendix J). Fresh continuations test each action pair from the same complete environment state, policy context, and remaining budget. Under the default selection rule, the preferred action achieves higher continuation success in 8 pairs, tied success in 5, and lower success in none. Its task-averaged advantage is 56.71 percentage points, with a 95% task-bootstrap interval of [28.47, 84.03]. The matched examples show that CPAD identifies local improvements even within trajectories already rewarded as successful, extracting guidance that terminal success alone cannot distinguish.

2. Paired token guidance makes more effective use of that action evidence (Section 4.4; Appendix K). On WebShop–1.5B, CPAD improves success nAUC by 6.19 points over direct empirical action advantages with the same selection rule and outcome anchor. Matching correction RMS isolates the contribution of pairing: CPAD retains nAUC gains of 17.39 and 42.44 points over positive-only and negative-only controls, with higher final success and lower reference KL. Together with the token-averaging and shuffling controls, these results establish the learning value of the paired token signal beyond action-level supervision and correction magnitude.

3. The mechanism delivers gains over strong baselines in both learning efficiency and task success (Section 4.5; Appendices F and L). On WebShop–7B, CPAD exceeds GraphGPO by 11.86 nAUC points over 150 updates, 18.89 points within the common 13.84-hour training budget, and 3.8 percentage points in final success. CPAD also achieves higher common-time success AUC than HGPO, GiGPO, and GraphGPO across all six ALFWorld/WebShop configurations, including privileged rescoring in the cost. Across two independent training seeds per model, its nAUC advantage over GraphGPO recurs at both WebShop–1.5B (+2.83 / +2.89 points) and 3B (+8.52 / +15.93 points), with final-success gains at 3B also reproduced (+0.6 / +3.0 percentage points).

4. Sparse guidance remains effective and extends to visual planning (Appendices D and L). On WebShop–3B, CPAD exceeds GRPO and GraphGPO in both nAUC and final success while correcting 7.99% of decisions on average. A stricter support setting on WebShop–1.5B exceeds GRPO at 4.69% coverage; zero-selection batches use the matched GRPO objective (Section 3.3). On visual Sokoban with Qwen2.5-VL-3B, CPAD reaches 33.42 nAUC and 57.03% final success, compared with 17.86 and 25.00% for GRPO. The gains therefore extend from text-based embodied tasks and web-tool interaction to visual spatial planning.

The revision also states the token-value alignment condition and idealized update analyzed by Proposition I.2 next to the correction rule (Section 3.2), and positions CPAD relative to Token-Importance Guided Direct Preference Optimization (Section 5). Across the audits and learning experiments, the evidence supports a coherent contribution: CPAD extracts valuable local action comparisons from rollout outcomes and converts them into effective token guidance for more efficient agent learning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.