acceptodds
Under review as a conference paper at ICLR 2027

PEARL: Process-Experience Aware Reinforcement Learning for Multi-Turn Agents

Abstract

Reinforcement learning for multi-turn agents often relies on sparse outcome signals that indicate whether a trajectory succeeds, but provide limited guidance about which intermediate decisions should be corrected. This becomes especially problematic for failed trajectories, which may still contain many valid actions and therefore should not be treated as uniformly negative experience. We propose PEARL (Process-Experience Aware Reinforcement Learning), a framework that learns from recurring failure patterns across past interactions. Rather than penalizing an entire failed trajectory, PEARL identifies failure patterns that repeatedly appear in unsuccessful behavior and uses successful experience to remove patterns that are not uniquely associated with failure. The remaining failure-specific patterns are then used to provide more targeted negative supervision during reinforcement learning. On AppWorld, PEARL+ARPO consistently improves over ARPO on both Normal and Challenge tasks at the 4B and 8B scales. On ALFWorld, PEARL achieves 97.9% mean success at 4B, outperforming the strongest baseline by 2.7 points, while remaining competitive at 8B. Extensive ablations and process analyses further validate the effectiveness of PEARL's selective failure-memory design.

Primary area:
reinforcement learning
Keywords:
LLM, Agentic rl

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_THkrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PEARL addresses credit assignments in failed agent trajectories, which may contain many useful actions alongside consequential mistakes. Built on ARPO’s branching rollouts, it compares continuations from a shared history that receive different terminal rewards and identifies their first token divergence. It represents the chosen tokens using its logits across the model’s later layers, clusters these representations into recent failure and success memories, and removes failure clusters that overlap with successful behavior. Matching decisions then receive stronger negative advantages from the divergence to the end of the action, while positive advantages remain unchanged. Across three training seeds, PEARL improves over ARPO on AppWorld at both model sizes and achieves 97.9% mean success on ALFWorld at 4B; at 8B, it improves over ARPO but trails GiGPO.

Critical strengths

The central insight is useful: successful experience can help determine which apparent failures should not receive additional punishment. This is more selective than simply penalizing frequently observed behavior in unsuccessful trajectories. The method requires neither manually labeled process supervision nor a separately trained reward model, and its memory operates only during training. The experimental controls are quite strong: main results include three-seed means and standard deviations, baselines use the same model/environment infrastructure, and the random-shaping ablation preserves the shaping budget and multipliers. That control supports the value of where negative weight is applied, beyond merely increasing negative updates. Ablations examine filtering, memory duration, representation, and shaping direction; concrete examples illustrate both correctly vetoed benign actions and retained invalid actions. The paper also appropriately acknowledges that its patterns indicate outcome association rather than proven causal errors.

Critical weaknesses

The most significant weaknses is the lack of direct comparison with MEDS, the explicitly acknowledged precursor that already combines intermediate representations, clustering, historical failure recurrence, and stronger penalties; without a matched adaptation, the incremental value of PEARL’s combined changes is insufficiently isolated. Branch comparisons also do not establish that the first divergent token—or the remainder of that action—caused the outcome difference, since subsequent decisions vary too. The case studies help, but there is no systematic audit of cluster purity or false penalties, and a failure centroid’s absence from a success cluster is weak evidence that the whole cluster is failure-specific. Robustness remains uncertain: ablations use single runs, changing clustering settings reduces hard-filter performance by over ten points, and AppWorld Challenge improvements over ARPO are only 1.16–1.40 points against substantial seed variability. Finally, the paper does not report runtime overhead or compute-matched comparisons, and it should clarify how hyperparameters were selected independently of the AppWorld test splits used throughout the ablations.

Questions for authors

Comparison with MEDS: What improvement does PEARL provide over a matched ARPO + MEDS baseline? Can you isolate the benefits of divergence localization and success-bank filtering?

3 comments
AuthorsOpenReview ↗

Response to reviewer THkr(1/3)

The most significant weaknses is the lack of direct comparison with MEDS, the explicitly acknowledged precursor that already combines intermediate representations, clustering, historical failure recurrence, and stronger penalties; without a matched adaptation, the incremental value of PEARL’s combined changes is insufficiently isolated.

Comparison with MEDS: What improvement does PEARL provide over a matched ARPO + MEDS baseline?

Thank you for the suggestion. MEDS targets single-turn mathematical reasoning, whereas PEARL targets multi-turn interaction with an environment. Consequently, MEDS cannot be directly used as a baseline on our tasks. MEDS extracts representations from the first token of the final answer and penalizes the reward of the entire response. In AppWorld and ALFWorld, success depends on a sequence of actions that changes the environment state. The final textual response does not reliably represent these decisions, and a failed trajectory can contain many correct actions.

Applying MEDS to these tasks would therefore require additional designs for where to extract action representations, how to label intermediate decisions, and which action tokens to penalize. The resulting comparison would evaluate a newly designed agent adaptation whose performance depends on these choices. These are precisely the problems PEARL addresses through shared-history branch comparisons, divergence-based localization, and success-based filtering for selective negative-advantage shaping. Sharing a representation technique does not make the two methods directly interchangeable.

Can you isolate the benefits of divergence localization and success-bank filtering?

Thank you for the suggestion. We evaluate these two components through separate controlled comparisons.

Divergence localization. We added feature-position ablations on AppWorld with Qwen3-4B, keeping success-bank filtering and the shaping region unchanged. We compare features extracted at the first actual divergence with features from a random token within the current action or within a randomly selected later step. The training seed is fixed, and the random-position results are averaged across multiple position-selection seeds at training step 400.

Feature extraction positionNormalChallenge
First actual divergence token (PEARL)57.74%31.02%
Random token within the current step56.55%29.30%
Random token within a randomly selected later step52.98%28.22%

Features at the first divergence outperform random features within the current step by 1.19/1.72 percentage points on Normal/Challenge, and features from later steps by 4.76/2.80 points. Random features within the current step still perform reasonably well, suggesting that useful decision information is also present elsewhere in the same action. These comparisons specifically isolate the benefit of extracting features at the divergence point while preserving where shaping is applied.

Success-bank filtering. We directly evaluate this component in §4.3, Fig. 4(b). With all other settings held constant, enabling the filter improves AppWorld success rates by 5.16 and 3.04 percentage points on Normal and Challenge, respectively. A decision in a failed trajectory is not necessarily responsible for the failure, especially when similar decisions also occur in successful trajectories. The success bank identifies these shared patterns and excludes them from additional penalties, reducing the risk of penalizing useful behavior.

Together, these comparisons support distinct contributions: divergence-aligned features help represent the decision being shaped, while success-bank filtering helps avoid additional penalties on patterns also associated with success. We will clarify this distinction and include the new localization results in the revised paper.

AuthorsOpenReview ↗

Response to reviewer THkr(2/3)

Branch comparisons also do not establish that the first divergent token—or the remainder of that action—caused the outcome difference, since subsequent decisions vary too.

Thank you for pointing this out. We agree that subsequent decisions also affect the final outcome, so branch comparisons cannot strictly establish that the first divergent token or its corresponding action caused the outcome difference. Our goal is to combine several signals to identify informative decision points and provide targeted training feedback.

First, an entropy increase indicates greater uncertainty in the model’s predictions: the model is less certain about how to continue, making this a potentially informative location for exploring alternative decisions. Second, if continuations branching from this location reach different outcomes, this provides an additional indication that choices around this point may be relevant to success or failure. We then locate the first token where the continuations actually differ and extract the corresponding decision features. Finally, success-bank filtering excludes failure patterns that also appear in successful experience, reducing the risk of treating useful behavior as a cause of failure.

Together, model uncertainty, contrasting branch outcomes, and successful experience guide the selection of informative decision points for training.

Our additional feature-position ablations evaluate the training value of these candidates. On AppWorld with Qwen3-4B, we keep success-bank filtering and the shaping region unchanged and vary only the feature extraction position. We fix the training seed and use multiple position-selection seeds for the random-position settings. The table reports results at training step 400, averaged across position-selection seeds for those settings.

Feature extraction positionNormalChallenge
First actual divergence token (PEARL)57.74%31.02%
Random token within the current step56.55%29.30%
Random token within a randomly selected later step52.98%28.22%

The first-divergence representation performs best on both splits. Random positions within the current step achieve relatively close results, while features from later steps lead to a further decline. These findings support the usefulness of features associated with the current divergence as a learning signal. These results support the practical training value of our decision-point selection. We will include this explanation and the additional results in the revised paper.

The case studies help, but there is no systematic audit of cluster purity or false penalties, and a failure centroid’s absence from a success cluster is weak evidence that the whole cluster is failure-specific.

Thank you for the suggestion. PEARL does not identify error patterns solely because a failure centroid is absent from the success clusters. We first compare two branches that share the same prefix but reach different outcomes, extract features at their first actual divergence, and then use success-bank filtering to exclude patterns that overlap with known successful behavior.

We agree that these two steps do not guarantee that every retained cluster member represents an erroneous action. PEARL aims to use these patterns to improve training, rather than precisely determine whether every individual action is correct or incorrect. Its effectiveness should be assessed through final task performance, while false penalties require a separate audit; illustrative case studies cannot replace that validation. We will clarify this distinction and revise terms such as “pure failure clusters” that could incorrectly imply the absence of false positives.

AuthorsOpenReview ↗

Response to reviewer THkr(3/3)

Robustness remains uncertain: ablations use single runs, changing clustering settings reduces hard-filter performance by over ten points, and AppWorld Challenge improvements over ARPO are only 1.16–1.40 points against substantial seed variability.

Thank you for raising these concerns about robustness.

(1) Our main experiments use three training seeds. Due to computational constraints, we have not yet repeated all ablation settings across multiple training seeds. We will make every effort to add runs with different random seeds and report means and standard deviations to better assess the consistency of each component’s benefits.

(2) MCS5/MS3 lowers the minimum cluster size and density requirements, allowing finer-grained clusters with fewer supporting samples. Since hard filtering retains or removes entire failure clusters, these smaller clusters can make filtering more sensitive to the local sample distribution. Table 2 shows that, under the same MCS5/MS3 setting, soft filtering achieves 53.57%/34.61% on Normal/Challenge, compared with 47.22%/20.22% for hard filtering. This comparison suggests that the performance drop reflects an interaction between finer-grained clustering and hard filtering, rather than finer-grained clustering being inherently ineffective. We will prioritize additional seeds to assess the stability of these differences.

(3) Our analysis of training and evaluation trajectories links the smaller gains on AppWorld Challenge to unfamiliar API usage and insufficient interaction budgets, beyond local decision errors.

Across three seeds, 64%–69% of API calls on Challenge involve usage patterns absent from the corresponding recent training trajectories, compared with 13%–18% on Normal. The model therefore needs to transfer its learned corrective experience to different tasks and interfaces. Furthermore, among failed trajectories, 31%–43% on Challenge reach the 40-step interaction limit, compared with 17%–28% on Normal. Even when individual decisions improve, a task can still fail if the model cannot complete it within the available steps.

Thus, harder tasks do not necessarily imply larger gains from PEARL. Unfamiliar API usage and limited interaction budgets can prevent local decision improvements from translating into task success. These findings help explain the smaller gains on Challenge; we will include this analysis in the revision.

Finally, the paper does not report runtime overhead or compute-matched comparisons, and it should clarify how hyperparameters were selected independently of the AppWorld test splits used throughout the ablations.

Thank you for requesting clarification on efficiency and hyperparameters.

Computational overhead. At the algorithmic level, PEARL can reuse intermediate hidden states from the standard training forward pass to construct decision features, without rerunning the Transformer or backpropagating through the feature extraction branch. For Qwen3-4B, we use the last 18 layers. At each layer, we compute a dot product between the 2,560-dimensional hidden state and the LM-head weight row for the selected token. Counting one multiplication and one addition per dimension as approximately 2 FLOPs, each feature requires approximately 18 × 2 × 2,560 = 92,160 additional FLOPs.

For comparison, a standard full-vocabulary projection scores all 151,936 candidate tokens to compute the next-token distribution, requiring approximately 2 × 2,560 × 151,936 = 777,912,320 FLOPs per position. The trajectory token is already known during feature extraction, so PEARL computes only its score at each selected layer. These dot products cost approximately 0.01185% of a full-vocabulary projection at one position. For N decision features and T full-vocabulary token projections per training step, the ratio is 0.01185% × N/T. Since a complete forward pass also includes the Transformer layers, this comparison illustrates the small arithmetic cost of the feature dot products.

Matching and clustering use 18-dimensional features. We retain experience from the most recent 25 updates and recluster every 10 updates. Across the three seeds analyzed for each of AppWorld and ALFWorld, the largest observed individual success or failure bank contained 154 features, keeping the data processed by these operations small. These arithmetic counts do not establish end-to-end runtime overhead or replace a compute-matched comparison. We will supplement them with runtime measurements and make every effort to add comparisons under the same total compute budget.

Hyperparameter selection. We did not use AppWorld test-set results to select hyperparameters. The default configuration was fixed before the test-set ablations, which were conducted to analyze the effects of different settings rather than select the best-performing configuration. We will clarify hyperparameter selection and the purpose of these ablations in the revision.

Reviewer_qebxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to assign credit in long horizon agentic tasks. The procedure follows ARPO (https://arxiv.org/abs/2507.19849), while leveraging more information in failed trajectories. The authors maintain a memory of recurring failure-associated decision patterns and selectively amplify negative advantages for the remaining patterns. Empirically, it has some improvement on AppWorld and ALFWorld.

Critical strengths
  • The proposed method shows improvement in AppWorld and ALFWorld, which is important long-horizon agentic tasks.
Critical weaknesses
  • The reason or intuition behind the design is not clear. The author simply introduce the method without telling any reason for this design. So I was not convinced that the proposed method will work in other tasks and it seems the authors are just combing some modules.
  • Novelty. I was not convinced that this paper significantly differs from The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping (https://arxiv.org/abs/2604.11297), and authors didn't compare with this paper as well.
  • The writing needs improvement. It takes me a long time to understand what is the porposed method. I highly recommend authors to organize section 3 (method) to make it clear.
Questions for authors
  1. Could the authors provide some reason/intutition behind the design? I was expecting authors to give some explanation/intutition why this method works well. For example, could authors show that the first branching token is very important to the performance?
  2. I noticed that in figure 3, the proposed mehtod performs worse than ARPO in the early stage, could the author clarify why that happens?
  3. Could the authors include an experimental comparison with The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping (https://arxiv.org/abs/2604.11297)? Given the similarities between the two methods, such a comparison would help clarify the empirical contribution of PEARL.
2 comments
AuthorsOpenReview ↗

Response to reviewer qebx(1/2)

The reason or intuition behind the design is not clear. The author simply introduce the method without telling any reason for this design. So I was not convinced that the proposed method will work in other tasks and it seems the authors are just combing some modules.

Could the authors provide some reason/intutition behind the design? I was expecting authors to give some explanation/intutition why this method works well. For example, could authors show that the first branching token is very important to the performance?

Thank you for raising this question. We apologize for not sufficiently explaining the motivation behind the overall design. PEARL’s core idea is to compare two continuations that share the same interaction history but reach different outcomes, identify local decisions worth correcting, and use this experience to guide subsequent training. Our intuition is that the point where these continuations first make different choices is likely to be a key decision affecting success or failure. We therefore extract decision features at the first actually differing token, use memory to identify recurring patterns that may lead to failure, and apply targeted penalties to the corresponding decision regions. The contrasting outcomes, decision features, memory, and shaping serve a common purpose: locating potentially problematic decisions and learning from patterns that recur across attempts. This rationale does not depend on task-specific API or action names, but on access to continuations with shared histories and different outcomes.

For feature extraction specifically, the choices at the first divergence are made under exactly the same context, so their difference more directly reflects different policy choices. Later tokens are conditioned on already-diverged prefixes, and their differences may also reflect wording, grammar, or semantic coherence. This motivates our choice of the first actual divergence token to represent the decision.

To test this choice, we added two ablations on AppWorld with Qwen3-4B: (1) randomly selecting a token after the first divergence within the current action, excluding the first differing token; and (2) randomly selecting a later step and then a token within that step. Both ablations change only the feature extraction position; shaping still applies to the original decision region in the current step. We fixed the training seed and ran the ablations with multiple position-selection seeds. The table reports results at training step 400, averaged across position-selection seeds.

Feature extraction positionNormalChallenge
First actual divergence token (PEARL)57.74%31.02%
Random token within the current step56.55%29.30%
Random token within a randomly selected later step52.98%28.22%

The first-divergence representation achieves 57.74%/31.02% on Normal/Challenge, outperforming both random-position alternatives and providing empirical support for this choice. Random selection within the current step achieves relatively close results of 56.55%/29.30%, suggesting that other positions within the same action still retain useful decision information. Using features from a later random step further reduces performance to 52.98%/28.22%, supporting the importance of aligning the extracted features with the decision being shaped.

These findings are consistent with our intuition that extracting features at the first divergence reduces interference from subsequent wording and context changes. We appreciate the reviewer’s suggestion to clarify and examine this design, and will include the overall motivation, these experiments, and the corresponding conclusions in the revised paper.

AuthorsOpenReview ↗

Response to reviewer qebx(2/2)

Novelty. I was not convinced that this paper significantly differs from The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping (https://arxiv.org/abs/2604.11297), and authors didn't compare with this paper as well.

Could the authors include an experimental comparison with The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping (https://arxiv.org/abs/2604.11297)? Given the similarities between the two methods, such a comparison would help clarify the empirical contribution of PEARL.

Thank you for the suggestion. MEDS targets single-turn mathematical reasoning, whereas PEARL targets multi-turn interaction with an environment. Consequently, MEDS cannot be directly used as a baseline on our tasks. MEDS extracts representations from the first token of the final answer and penalizes the reward of the entire response. In AppWorld and ALFWorld, success depends on a sequence of actions that changes the environment state. The final textual response does not reliably represent these decisions, and a failed trajectory can contain many correct actions.

Applying MEDS to these tasks would therefore require additional designs for where to extract action representations, how to label intermediate decisions, and which action tokens to penalize. The resulting comparison would evaluate a newly designed agent adaptation whose performance depends on these choices. These are precisely the problems PEARL addresses through shared-history branch comparisons, divergence-based localization, and success-based filtering for selective negative-advantage shaping. Sharing a representation technique does not make the two methods directly interchangeable.

The writing needs improvement. It takes me a long time to understand what is the porposed method. I highly recommend authors to organize section 3 (method) to make it clear.

Thank you for this feedback. We apologize that Section 3 does not explain the method clearly enough. Intuitively, PEARL learns from contrasting continuations: (1) we collect parent–branch pairs that share the same interaction history but reach different outcomes; (2) we locate their first differing token and extract decision features that describe the choices made there; (3) we store these features in memory and cluster them to identify recurring failure patterns; and (4) we use these patterns to selectively strengthen penalties on the corresponding decision regions during training.

We will reorganize Section 3 around this workflow, explaining the intuition and purpose of each step before introducing its mathematical formulation. We will incorporate these changes into the camera-ready version if the paper is accepted.

I noticed that in figure 3, the proposed mehtod performs worse than ARPO in the early stage, could the author clarify why that happens?

Thank you for pointing this out. Early in training, ARPO updates the policy directly using trajectory rewards, whereas PEARL also applies penalties based on failure patterns stored in memory. With limited experience, these penalties can discourage decisions that fail initially but are still worth exploring, explaining why PEARL initially performs worse than ARPO. As more experience is collected, the failure patterns become more reliable, and the targeted penalties start to help. In future work, we may try warming up the memory before applying these penalties, giving PEARL more experience to learn from and potentially helping it improve faster.

Reviewer_Rb22OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

the adds a selective failure memory to ARPO. It treats parent–branch pairs with unequal rewards as local contrasts and encodes the first divergent decision as an 18-dimensional late-layer logit profile of the chosen token. Failure-side patterns that recur and aren’t vetoed by success-bank clusters receive amplified negative advantages. The paper reports consistent gains over ARPO on AppWorld and the best 4B result on ALFWorld.

Critical strengths

None

Critical weaknesses

I think the novelty is fairly limited. Most of the pieces already exist, the layer-logit fingerprint and clustering it for shaping come from MEDS, and the branching comes from ARPO. What PEARL really adds is using parent–branch pairs to label decisions, the success-bank veto, and keeping only recent memory. These are sensible tweaks, but since the paper never compares against MEDS directly, I can’t tell how much they actually help.

Questions for authors

The authors take ARPO’s entropy-based branching settings as they are, with no testing. That’s a problem, because where the branches happen decides which pairs end up in the memory. I’d like to know why these values were chosen and how much the results change if they’re adjusted. On the Challenge split, PEARL only gains about a point, which is within the seed variance. That surprised me, since hard multi-step tasks are exactly where better credit assignment should pay off most. I’d like the authors to explain this, maybe by showing how often the shaping actually kicks in on Challenge tasks compared to Normal ones.

2 comments
AuthorsOpenReview ↗

Response to reviewer Rb22(1/2)

I think the novelty is fairly limited. Most of the pieces already exist, the layer-logit fingerprint and clustering it for shaping come from MEDS, and the branching comes from ARPO. What PEARL really adds is using parent–branch pairs to label decisions, the success-bank veto, and keeping only recent memory. These are sensible tweaks, but since the paper never compares against MEDS directly, I can’t tell how much they actually help.

Thank you for the suggestion. MEDS targets single-turn mathematical reasoning, whereas PEARL targets multi-turn interaction with an environment. Consequently, MEDS cannot be directly used as a baseline on our tasks. MEDS extracts representations from the first token of the final answer and penalizes the reward of the entire response. In AppWorld and ALFWorld, success depends on a sequence of actions that changes the environment state. The final textual response does not reliably represent these decisions, and a failed trajectory can contain many correct actions.

Applying MEDS to these tasks would therefore require additional designs for where to extract action representations, how to label intermediate decisions, and which action tokens to penalize. The resulting comparison would evaluate a newly designed agent adaptation whose performance depends on these choices. These are precisely the problems PEARL addresses through shared-history branch comparisons, divergence-based localization, and success-based filtering for selective negative-advantage shaping. Sharing a representation technique does not make the two methods directly interchangeable.

The authors take ARPO’s entropy-based branching settings as they are, with no testing. That’s a problem, because where the branches happen decides which pairs end up in the memory. I’d like to know why these values were chosen and how much the results change if they’re adjusted.

Thank you for pointing this out. We initially retained ARPO’s branching settings to keep the underlying sampling procedure consistent and assess the contribution of PEARL’s memory and shaping mechanisms. We agree that branching locations affect which contrasting outcomes enter the memory, and have therefore added a branching-threshold sensitivity study.

On AppWorld with Qwen3-4B, we used training seed 2025 and trained for 400 steps, changing only the branching threshold from the default of 0.5 to 0.3 or 0.7:

Branching thresholdNormalChallengeAverage success rate
0.356.55%28.70%42.62%
0.5 (default)57.74%31.02%44.38%
0.750.20%25.74%37.97%

Lowering the threshold to 0.3 reduces average success by approximately 1.8 percentage points, whereas raising it to 0.7 produces a larger drop of 6.41 points. These results show sensitivity to the branching threshold, particularly when branching becomes more restrictive. A higher threshold makes branches harder to trigger and may limit the contrasting outcomes available for memory construction, affecting subsequent shaping. This is consistent with the reviewer’s concern that branching settings influence the experience collected by PEARL.

The default threshold of 0.5 performs best among the three tested settings. This configuration was fixed before the additional experiments. We will include this threshold sensitivity analysis in the revised paper.

AuthorsOpenReview ↗

Response to reviewer Rb22(2/2)

On the Challenge split, PEARL only gains about a point, which is within the seed variance. That surprised me, since hard multi-step tasks are exactly where better credit assignment should pay off most. I’d like the authors to explain this, maybe by showing how often the shaping actually kicks in on Challenge tasks compared to Normal ones.

Thank you for the suggestion. We further examined the training and evaluation trajectories and found that the difficulties on Challenge extend beyond local decision errors to unfamiliar API usage and insufficient interaction budgets. These observations help explain why improved credit assignment does not necessarily translate into larger gains on Challenge.

First, many API usage patterns in Challenge do not appear in the corresponding recent training trajectories. Across three seeds, such calls account for 64%–69% of API calls on Challenge, compared with 13%–18% on Normal. The model therefore needs to transfer its learned corrective experience to different tasks and interfaces, rather than directly reuse experience that covers every situation.

Second, Challenge trajectories are more likely to exhaust the interaction budget before completing the task. Among failed trajectories, 31%–43% on Challenge reach the 40-step limit, compared with 17%–28% on Normal. Even when individual decisions improve, a task can still fail if the model cannot complete it within the available steps.

Thus, harder tasks do not necessarily imply larger gains from PEARL. PEARL helps the model learn from decision errors during training, but unfamiliar API usage and limited interaction budgets can prevent local improvements from translating into task success. These trajectory-level findings help explain the smaller average gains on Challenge, and we will include this analysis in the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.