- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
PEARL addresses credit assignments in failed agent trajectories, which may contain many useful actions alongside consequential mistakes. Built on ARPO’s branching rollouts, it compares continuations from a shared history that receive different terminal rewards and identifies their first token divergence. It represents the chosen tokens using its logits across the model’s later layers, clusters these representations into recent failure and success memories, and removes failure clusters that overlap with successful behavior. Matching decisions then receive stronger negative advantages from the divergence to the end of the action, while positive advantages remain unchanged. Across three training seeds, PEARL improves over ARPO on AppWorld at both model sizes and achieves 97.9% mean success on ALFWorld at 4B; at 8B, it improves over ARPO but trails GiGPO.
Critical strengths
The central insight is useful: successful experience can help determine which apparent failures should not receive additional punishment. This is more selective than simply penalizing frequently observed behavior in unsuccessful trajectories. The method requires neither manually labeled process supervision nor a separately trained reward model, and its memory operates only during training. The experimental controls are quite strong: main results include three-seed means and standard deviations, baselines use the same model/environment infrastructure, and the random-shaping ablation preserves the shaping budget and multipliers. That control supports the value of where negative weight is applied, beyond merely increasing negative updates. Ablations examine filtering, memory duration, representation, and shaping direction; concrete examples illustrate both correctly vetoed benign actions and retained invalid actions. The paper also appropriately acknowledges that its patterns indicate outcome association rather than proven causal errors.
Critical weaknesses
The most significant weaknses is the lack of direct comparison with MEDS, the explicitly acknowledged precursor that already combines intermediate representations, clustering, historical failure recurrence, and stronger penalties; without a matched adaptation, the incremental value of PEARL’s combined changes is insufficiently isolated. Branch comparisons also do not establish that the first divergent token—or the remainder of that action—caused the outcome difference, since subsequent decisions vary too. The case studies help, but there is no systematic audit of cluster purity or false penalties, and a failure centroid’s absence from a success cluster is weak evidence that the whole cluster is failure-specific. Robustness remains uncertain: ablations use single runs, changing clustering settings reduces hard-filter performance by over ten points, and AppWorld Challenge improvements over ARPO are only 1.16–1.40 points against substantial seed variability. Finally, the paper does not report runtime overhead or compute-matched comparisons, and it should clarify how hyperparameters were selected independently of the AppWorld test splits used throughout the ablations.
Questions for authors
Comparison with MEDS: What improvement does PEARL provide over a matched ARPO + MEDS baseline? Can you isolate the benefits of divergence localization and success-bank filtering?
3 comments
Response to reviewer THkr(2/3)
Branch comparisons also do not establish that the first divergent token—or the remainder of that action—caused the outcome difference, since subsequent decisions vary too.
Thank you for pointing this out. We agree that subsequent decisions also affect the final outcome, so branch comparisons cannot strictly establish that the first divergent token or its corresponding action caused the outcome difference. Our goal is to combine several signals to identify informative decision points and provide targeted training feedback.
First, an entropy increase indicates greater uncertainty in the model’s predictions: the model is less certain about how to continue, making this a potentially informative location for exploring alternative decisions. Second, if continuations branching from this location reach different outcomes, this provides an additional indication that choices around this point may be relevant to success or failure. We then locate the first token where the continuations actually differ and extract the corresponding decision features. Finally, success-bank filtering excludes failure patterns that also appear in successful experience, reducing the risk of treating useful behavior as a cause of failure.
Together, model uncertainty, contrasting branch outcomes, and successful experience guide the selection of informative decision points for training.
Our additional feature-position ablations evaluate the training value of these candidates. On AppWorld with Qwen3-4B, we keep success-bank filtering and the shaping region unchanged and vary only the feature extraction position. We fix the training seed and use multiple position-selection seeds for the random-position settings. The table reports results at training step 400, averaged across position-selection seeds for those settings.
| Feature extraction position | Normal | Challenge |
|---|---|---|
| First actual divergence token (PEARL) | 57.74% | 31.02% |
| Random token within the current step | 56.55% | 29.30% |
| Random token within a randomly selected later step | 52.98% | 28.22% |
The first-divergence representation performs best on both splits. Random positions within the current step achieve relatively close results, while features from later steps lead to a further decline. These findings support the usefulness of features associated with the current divergence as a learning signal. These results support the practical training value of our decision-point selection. We will include this explanation and the additional results in the revised paper.
The case studies help, but there is no systematic audit of cluster purity or false penalties, and a failure centroid’s absence from a success cluster is weak evidence that the whole cluster is failure-specific.
Thank you for the suggestion. PEARL does not identify error patterns solely because a failure centroid is absent from the success clusters. We first compare two branches that share the same prefix but reach different outcomes, extract features at their first actual divergence, and then use success-bank filtering to exclude patterns that overlap with known successful behavior.
We agree that these two steps do not guarantee that every retained cluster member represents an erroneous action. PEARL aims to use these patterns to improve training, rather than precisely determine whether every individual action is correct or incorrect. Its effectiveness should be assessed through final task performance, while false penalties require a separate audit; illustrative case studies cannot replace that validation. We will clarify this distinction and revise terms such as “pure failure clusters” that could incorrectly imply the absence of false positives.
Response to reviewer THkr(3/3)
Robustness remains uncertain: ablations use single runs, changing clustering settings reduces hard-filter performance by over ten points, and AppWorld Challenge improvements over ARPO are only 1.16–1.40 points against substantial seed variability.
Thank you for raising these concerns about robustness.
(1) Our main experiments use three training seeds. Due to computational constraints, we have not yet repeated all ablation settings across multiple training seeds. We will make every effort to add runs with different random seeds and report means and standard deviations to better assess the consistency of each component’s benefits.
(2) MCS5/MS3 lowers the minimum cluster size and density requirements, allowing finer-grained clusters with fewer supporting samples. Since hard filtering retains or removes entire failure clusters, these smaller clusters can make filtering more sensitive to the local sample distribution. Table 2 shows that, under the same MCS5/MS3 setting, soft filtering achieves 53.57%/34.61% on Normal/Challenge, compared with 47.22%/20.22% for hard filtering. This comparison suggests that the performance drop reflects an interaction between finer-grained clustering and hard filtering, rather than finer-grained clustering being inherently ineffective. We will prioritize additional seeds to assess the stability of these differences.
(3) Our analysis of training and evaluation trajectories links the smaller gains on AppWorld Challenge to unfamiliar API usage and insufficient interaction budgets, beyond local decision errors.
Across three seeds, 64%–69% of API calls on Challenge involve usage patterns absent from the corresponding recent training trajectories, compared with 13%–18% on Normal. The model therefore needs to transfer its learned corrective experience to different tasks and interfaces. Furthermore, among failed trajectories, 31%–43% on Challenge reach the 40-step interaction limit, compared with 17%–28% on Normal. Even when individual decisions improve, a task can still fail if the model cannot complete it within the available steps.
Thus, harder tasks do not necessarily imply larger gains from PEARL. Unfamiliar API usage and limited interaction budgets can prevent local decision improvements from translating into task success. These findings help explain the smaller gains on Challenge; we will include this analysis in the revision.
Finally, the paper does not report runtime overhead or compute-matched comparisons, and it should clarify how hyperparameters were selected independently of the AppWorld test splits used throughout the ablations.
Thank you for requesting clarification on efficiency and hyperparameters.
Computational overhead. At the algorithmic level, PEARL can reuse intermediate hidden states from the standard training forward pass to construct decision features, without rerunning the Transformer or backpropagating through the feature extraction branch. For Qwen3-4B, we use the last 18 layers. At each layer, we compute a dot product between the 2,560-dimensional hidden state and the LM-head weight row for the selected token. Counting one multiplication and one addition per dimension as approximately 2 FLOPs, each feature requires approximately 18 × 2 × 2,560 = 92,160 additional FLOPs.
For comparison, a standard full-vocabulary projection scores all 151,936 candidate tokens to compute the next-token distribution, requiring approximately 2 × 2,560 × 151,936 = 777,912,320 FLOPs per position. The trajectory token is already known during feature extraction, so PEARL computes only its score at each selected layer. These dot products cost approximately 0.01185% of a full-vocabulary projection at one position. For N decision features and T full-vocabulary token projections per training step, the ratio is 0.01185% × N/T. Since a complete forward pass also includes the Transformer layers, this comparison illustrates the small arithmetic cost of the feature dot products.
Matching and clustering use 18-dimensional features. We retain experience from the most recent 25 updates and recluster every 10 updates. Across the three seeds analyzed for each of AppWorld and ALFWorld, the largest observed individual success or failure bank contained 154 features, keeping the data processed by these operations small. These arithmetic counts do not establish end-to-end runtime overhead or replace a compute-matched comparison. We will supplement them with runtime measurements and make every effort to add comparisons under the same total compute budget.
Hyperparameter selection. We did not use AppWorld test-set results to select hyperparameters. The default configuration was fixed before the test-set ablations, which were conducted to analyze the effects of different settings rather than select the best-performing configuration. We will clarify hyperparameter selection and the purpose of these ablations in the revision.
Response to reviewer THkr(1/3)
Thank you for the suggestion. MEDS targets single-turn mathematical reasoning, whereas PEARL targets multi-turn interaction with an environment. Consequently, MEDS cannot be directly used as a baseline on our tasks. MEDS extracts representations from the first token of the final answer and penalizes the reward of the entire response. In AppWorld and ALFWorld, success depends on a sequence of actions that changes the environment state. The final textual response does not reliably represent these decisions, and a failed trajectory can contain many correct actions.
Applying MEDS to these tasks would therefore require additional designs for where to extract action representations, how to label intermediate decisions, and which action tokens to penalize. The resulting comparison would evaluate a newly designed agent adaptation whose performance depends on these choices. These are precisely the problems PEARL addresses through shared-history branch comparisons, divergence-based localization, and success-based filtering for selective negative-advantage shaping. Sharing a representation technique does not make the two methods directly interchangeable.
Thank you for the suggestion. We evaluate these two components through separate controlled comparisons.
Divergence localization. We added feature-position ablations on AppWorld with Qwen3-4B, keeping success-bank filtering and the shaping region unchanged. We compare features extracted at the first actual divergence with features from a random token within the current action or within a randomly selected later step. The training seed is fixed, and the random-position results are averaged across multiple position-selection seeds at training step 400.
Features at the first divergence outperform random features within the current step by 1.19/1.72 percentage points on Normal/Challenge, and features from later steps by 4.76/2.80 points. Random features within the current step still perform reasonably well, suggesting that useful decision information is also present elsewhere in the same action. These comparisons specifically isolate the benefit of extracting features at the divergence point while preserving where shaping is applied.
Success-bank filtering. We directly evaluate this component in §4.3, Fig. 4(b). With all other settings held constant, enabling the filter improves AppWorld success rates by 5.16 and 3.04 percentage points on Normal and Challenge, respectively. A decision in a failed trajectory is not necessarily responsible for the failure, especially when similar decisions also occur in successful trajectories. The success bank identifies these shared patterns and excludes them from additional penalties, reducing the risk of penalizing useful behavior.
Together, these comparisons support distinct contributions: divergence-aligned features help represent the decision being shaped, while success-bank filtering helps avoid additional penalties on patterns also associated with success. We will clarify this distinction and include the new localization results in the revised paper.