acceptodds
Under review as a conference paper at ICLR 2027

When History Fails to Become Experience: Action Calibration in Language Agents

Abstract

Language agents should draw on prior attempts and environmental feedback to improve subsequent decisions within the same task. However, providing additional interaction history can sometimes reduce task success, suggesting that agents do not consistently use this information effectively. To investigate this limitation, we conduct further experiments and find that, although history improves overall task completion, much of this benefit persists even when past actions are shuffled. Disrupting the correspondence between actions and observations causes only a modest decline in task success. We therefore hypothesize that agents do not reliably connect past actions with their outcomes when deciding how to proceed. To test this hypothesis, we explicitly label each returned observation as the outcome of the preceding action. This simple annotation improves task success and reduces next-action repetition without introducing new environmental information. Building on this insight, we introduce a learned calibrator that explicitly reassesses past experience and selectively records it to guide subsequent decisions, yielding further improvements in task success beyond outcome labeling alone.

Primary area:
generative models
Keywords:
LLM, experience, context

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_b3ZsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Two issues remain. A generic history-check reminder reaches 48.60% versus 48.25% for outcome labeling, so the gains may come from more careful history checking rather than action–outcome binding. The calibrator also adds a review call after each step, with labels from GPT judgments rather than executed outcomes, and no cost-matched comparison or independent label-quality check is provided.

Critical strengths
  1. Well-motivated experiments. The history interventions, fixed-history repetition analysis, and full rollouts connect the initial diagnosis to the proposed interventions. This makes the paper more informative than a method evaluated solely by its final success rate.
  2. Broad empirical validation. The calibrator improves over the raw-history baseline in every reported model-level and environment-level aggregate. The overall gain of 5.06 percentage points suggests that the approach is useful beyond a single actor or benchmark.
Critical weaknesses
  1. The proposed mechanism is not fully isolated. A generic history-check reminder reaches 48.60% success, comparable to the 48.25% achieved by explicitly labeling action outcomes (Appendix D.1). It remains unclear how much of the latter’s benefit comes from action–outcome attribution rather than prompting the actor to reconsider its history. This limits the strength of the paper’s central mechanistic claim.
  2. The calibrator’s benefit is not established under a matched inference budget. Unlike the raw-history and outcome-labeling conditions, the calibrator adds a review call after each completed nonterminal interaction. Its training targets also depend on model-generated candidates and model-based judgments. Without cost-matched results and an independent check of label quality, it is difficult to assess the practical significance of its improvement.
Questions for authors
  1. Under matched prompt length and inference budget, how do outcome labeling, the history-check reminder, and their combination compare? The answer would help distinguish action–outcome attribution from a more general history-rechecking effect.
  2. What are the average additional calls, tokens, and costs of the calibrator, and does its advantage persist under a matched budget? How well do independent or human assessments agree with the model-based judgments used to construct its training targets?
4 comments
AuthorsOpenReview ↗

Response to Reviewer b3Zs (part 1)

Thank you for recognizing the connection between the diagnostic experiments and the interventions. Below, we report the added combined condition and measured review overhead.

W1: The proposed mechanism is not fully isolated. A generic history-check reminder reaches 48.60% success, comparable to the 48.25% achieved by explicitly labeling action outcomes (Appendix D.1). It remains unclear how much of the latter’s benefit comes from action–outcome attribution rather than prompting the actor to reconsider its history. This limits the strength of the paper’s central mechanistic claim.

Response: Similar performance from Causal and Check is precisely our intended finding. Causal adds no review instruction: it labels observations as outcomes of preceding actions, implicitly prompting reassessment. Check requests reassessment explicitly, so similar performance is expected.

We further combine Causal and Check and compare the following four conditions on the same 2,000 tasks (100 per model–environment cell). Actor rows average four environments; Overall averages the five actors:

ActorNRaw (%)Causal (%)Check (%)Causal + Check (%)
GPT-5.6-Sol40074.2575.7576.7576.25
DeepSeek-V4-Pro40065.7568.5067.5068.00
Qwen3.5-Plus40053.7556.0056.0055.75
Qwen3-32B40030.0031.0029.7531.25
Qwen3-8B40017.0019.2518.5018.50
Overall2,00048.1550.1049.7049.95

Overall, both methods improve over Raw. Their combination does not outperform Causal alone. This is consistent with overlapping roles in history checking.

W2: The calibrator’s benefit is not established under a matched inference budget. Unlike the raw-history and outcome-labeling conditions, the calibrator adds a review call after each completed nonterminal interaction. Its training targets also depend on model-generated candidates and model-based judgments. Without cost-matched results and an independent check of label quality, it is difficult to assess the practical significance of its improvement.

Response: Calibrator is an optimization of Experience. Both generate experiences from history, then let the original actor choose actions using that history and the retained experiences. Calibrator trains a model to generate better experiences. The original method-evaluation figure did not display Experience alongside Calibrator. Figure 4 had already shown that Experience performed worse than Raw. This motivated us to train a better experience generator. Experience therefore provides a direct baseline for comparing review overhead and benefits. Each condition uses 2,000 tasks, with 100 per model–environment cell. Rates average the five actors and four environments:

MethodSuccess rate (%)
Raw48.15
Experience46.70
Calibrator53.70

We also compare their output-token use. For each actor, output tokens are averaged over four environments, with 100 tasks per environment. Cells report actor output + review-call output in k tokens per task. For example, 1.45 + 1.40 means 1.45k actor output tokens and 1.4k review-call output tokens per task.

ActorRawExperienceCalibrator
GPT-5.6-Sol1.45 + 0.001.45 + 1.401.48 + 0.21
DeepSeek-V4-Pro4.53 + 0.004.85 + 0.374.04 + 0.23
Qwen3.5-Plus4.25 + 0.004.24 + 0.214.03 + 0.23
Qwen3-32B1.59 + 0.001.72 + 0.241.64 + 0.23
Qwen3-8B2.59 + 0.002.54 + 0.482.57 + 0.26

Calibrator’s review-call output per task is substantially smaller than the actor’s output. Calibrator adds little output-token overhead. Its total output is usually similar to Raw and lower than Experience. Its 8B experience generator further reduces resource cost.

GPT-5.6-Sol’s larger review-call output may be related to internal reasoning tokens. These account for about 71.4% of its review-call output tokens pooled across the four environments. This explains its approximately 1.40k review-call output tokens per task. The other models generate both actions and experiences with thinking mode disabled.

AuthorsOpenReview ↗

Response to Reviewer b3Zs (part 2)

Q1: Under matched prompt length and inference budget, how do outcome labeling, the history-check reminder, and their combination compare? The answer would help distinguish action–outcome attribution from a more general history-rechecking effect.

Response: The prompts and inference budgets for Causal, Check, and their combination are already similar, with the prompts differing by only a limited number of tokens. We pool actor calls across models and environments. The per-call means below are total input or output tokens divided by total calls. The differences are small.

ConditionAverage input tokens per callAverage output tokens per call
Causal3,137.05177.57
Check3,281.84178.10
Causal + Check3,447.44190.12

For performance results, please see our response to W1 above. Overall, combining the two methods brings no additional advantage, further indicating that they operate through similar mechanisms.

Q2: What are the average additional calls, tokens, and costs of the calibrator, and does its advantage persist under a matched budget? How well do independent or human assessments agree with the model-based judgments used to construct its training targets?

Response: Output token statistics for each actor are provided in W2 above. To assess agreement with human judgments, we randomly selected 100 experiences for human assessment. The assessment was conducted without access to GPT’s judgments. Human and GPT judgments agree in 83% of cases, indicating that GPT-based data selection is reliable. The calibrator trained on these data also achieves clear performance improvements. These results show that the judgments are reliable.

Thank you for the additional experiments. The combined Causal and Check result is consistent with the two interventions having overlapping effects. The Experience baseline, token breakdown, and blind human check also help address my evaluation concerns. These additions strengthen the paper. I will update my score to weak accept.

AuthorsOpenReview ↗

Response to Reviewer b3Zs (Please update the score)

Thank you for your positive assessment of the additional experiments and for your constructive feedback throughout the discussion. We noticed that the displayed rating may not yet reflect your intended update to weak accept. Could you please update the rating in your original review accordingly?

Reviewer_RpSNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether language agents use past actions and environmental feedback to improve subsequent decisions. Across five models and four environments, the authors compare truncated histories, observation-only histories, and shuffled historical actions. History generally helps, but disrupting action–observation correspondence causes only a modest performance decline.

Critical strengths
  1. The evaluation covers five models and four environments, with 10,000 task executions per main condition. Supplementary experiments reproduce the main trends using separate user–assistant messages, reducing dependence on one history format.
  2. The paper distinguishes reduced repetition, better next actions, and eventual task completion. Self-generated experience reduces repetition without improving success, illustrating the limits of simple behavioral measures. The content-screening ablation also supports the contribution of action-comparison supervision.
Critical weaknesses
  1. History shuffling and outcome labeling do not uniquely identify a failure to associate actions with outcomes. Shuffle-All reduces success by only 2.55 points in Table 2, but preserving action strings may be unnecessary when observations already contain substantial state-change information. Meanwhile, the generic history-check reminder in Appendix D.1 achieves 48.60% success, comparable to 48.25% with outcome labeling. The benefit may therefore reflect more careful history checking rather than explicit action–outcome binding alone. The experiments establish sensitivity to presentation more directly than the proposed failure mechanism.
  2. The recovery experiment in Section 3 supplies a future segment from a failed trajectory rather than history actually executed after restoration. Although the prompt clearly separates reference and current state, the agent must reconcile actual history, an unexecuted reference branch, and a new continuation. The decline could partly reflect branch-state confusion or imitation of failed examples. Being below the context limit does not rule out interference from additional text. The reduction from 16.6% to 13.2% is informative for this reference-history setting, but should not directly establish a general failure to turn ordinary interaction history into experience.
  3. Calibrator labels depend on one model’s judgments of candidate experiences and next actions rather than executed environmental outcomes. Section 6.1 also retains experiences that leave every actor’s next action unchanged, on the assumption that they may help later; that delayed value is not verified. The content-screening ablation and final success evaluation support the full pipeline, but do not separate better experience generation, selective writing, and distillation of judge preferences.
  4. Section 6.4 changes both demonstration sources and output targets in the joint-distillation comparison. The action-only model uses successful trajectories from ordinary GPT-5.6-Sol, while the joint model uses successful trajectories generated with the calibrator. The latter may therefore receive different action demonstrations. Equal data amounts and five training seeds control some factors, but do not separate demonstration quality from generating calibration text.
Questions for authors

Please refer to weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer RpSN (Part 1)

Thank you for recognizing the behavioral analysis and action-comparison ablation. We add controlled experiments on observation content, recovery, and teacher demonstrations.

W1: History shuffling and outcome labeling do not uniquely identify a failure to associate actions with outcomes. Shuffle-All reduces success by only 2.55 points in Table 2, but preserving action strings may be unnecessary when observations already contain substantial state-change information. Meanwhile, the generic history-check reminder in Appendix D.1 achieves 48.60% success, comparable to 48.25% with outcome labeling. The benefit may therefore reflect more careful history checking rather than explicit action–outcome binding alone. The experiments establish sensitivity to presentation more directly than the proposed failure mechanism.

Response: The limited drop of Shuffle-All is exactly what we want to show. Observations indeed preserve substantial state-change information, explaining Observation-Only’s performance. Shuffle-All’s limited effect indicates that models rely more on observations than on historical actions and their pairings.

There are two possible explanations: action–outcome information lacks value, or the actor fails to use it effectively. Causal and Check improve success by promoting feedback use. These gains indicate that action–outcome information has value but is underused by the actor.

We also examine whether actors notice abnormalities in their history. GPT-5.4 classifies saved responses from shuffle rollouts, including visible thinking, to check whether actors recognize history mismatches. Rates pool individual responses across models and environments. Actors do not recognize mismatches in 97.91% of the 27,049 Shuffle-All responses. The rate is 99.06% for the Shuffle-Old responses. In most responses, the actor does not recognize the mismatch. This further indicates a problem in history use.

The similar gains from Causal and Check are precisely our point. Causal adds no review instruction: it labels observations as outcomes of preceding actions, implicitly prompting reassessment. Check requests reassessment explicitly, so similar performance is expected.

We compare the four conditions on the same 2,000 tasks (100 per model–environment cell). Actor rows average four environments; Overall averages the five actors:

ActorNRaw (%)Causal (%)Check (%)Causal + Check (%)
GPT-5.6-Sol40074.2575.7576.7576.25
DeepSeek-V4-Pro40065.7568.5067.5068.00
Qwen3.5-Plus40053.7556.0056.0055.75
Qwen3-32B40030.0031.0029.7531.25
Qwen3-8B40017.0019.2518.5018.50
Overall2,00048.1550.1049.7049.95

Overall, the combination does not exceed Causal alone. This is consistent with overlapping roles in history checking.

W2: The recovery experiment in Section 3 supplies a future segment from a failed trajectory rather than history actually executed after restoration. Although the prompt clearly separates reference and current state, the agent must reconcile actual history, an unexecuted reference branch, and a new continuation. The decline could partly reflect branch-state confusion or imitation of failed examples. Being below the context limit does not rule out interference from additional text. The reduction from 16.6% to 13.2% is informative for this reference-history setting, but should not directly establish a general failure to turn ordinary interaction history into experience.

Response: Actors rarely confuse reference and current state. GPT-5.4 checks the reference, actual interaction records, and the actor’s continuation reasoning and actions. It identifies assumptions that unexecuted reference events have already occurred in the current environment. Only 29 of 1,000 trajectories (2.90%) show such confusion.

ModelALFWorldAppWorldScienceWorldWebShopTotal
GPT-5.6-Sol00101
DeepSeek-V4-Pro01001
Qwen3.5-Plus02002
Qwen3-32B530311
Qwen3-8B930214
Overall1491529

We do not directly establish a general failure to turn ordinary interaction history into experience. This is a motivating phenomenon for understanding how models use contextual information.

We also compare Causal, Experience and Calibrator with and without reference on the same 1,000 sources. Rates average five actors and four environments, with 50 sources per cell. The results show that better use of history also improves the use of reference information.

MethodWithout reference (%)With reference (%)
Raw16.6013.20
Causal18.4018.10
Experience16.1017.80
Calibrator19.2019.90
AuthorsOpenReview ↗

Response to Reviewer RpSN (Part 2)

W3: Calibrator labels depend on one model’s judgments of candidate experiences and next actions rather than executed environmental outcomes. Section 6.1 also retains experiences that leave every actor’s next action unchanged, on the assumption that they may help later; that delayed value is not verified. The content-screening ablation and final success evaluation support the full pipeline, but do not separate better experience generation, selective writing, and distillation of judge preferences.

Response: Task success reflects the combined effects of many actions, so a final outcome cannot isolate the quality of one action or experience. Measuring one action’s effect would require many continuation runs from that point. We therefore use an LLM judge. The same issue makes step-level rewards difficult in agentic tasks, so LLM-based judging is widely used to assess action quality.

All-Tie experiences leave every actor’s next action unchanged. We compare retaining these training examples, removing them, or relabeling them as NO_UPDATE (no experience is written). Each scheme uses 2,000 tasks, with 100 per model–environment cell. Rates average five actors and four environments:

Treatment of All-Tie examplesSuccess rate (%)
Retain All-Tie53.70
Remove All-Tie51.25
Relabel All-Tie as NO_UPDATE51.40

Retaining All-Tie outperforms the other two schemes by 2.45 and 2.30 pp, respectively. This supports the training value of retaining such experiences, so that experience learning serves subsequent task progress rather than only immediate next-action improvement.

To further distinguish the contributions of experience generation and selective writing, we separate review into two components. “Whether to write” means choosing, at each review, between writing an experience and returning NO_UPDATE. “What to write” means generating the experience content after deciding to write. The full method uses Calibrator for both; the two hybrid conditions replace one component with GPT-5.6-Sol.

In the hybrid conditions, the model responsible for writing decisions first returns CALIBRATE or NO_UPDATE. Only a CALIBRATE decision triggers the other model to generate an experience from the same task and history. A NO_UPDATE decision writes nothing. We use only the first model’s writing decision, not its experience draft. The second model is instructed to generate only the experience content, without deciding again whether to write. We train a separate experience generator on the calibrator’s positive training examples.

Each condition uses 2,000 tasks, with 100 per model–environment cell. Rates average the five actors and four environments:

Writing decisionExperience generationSuccess rate (%)
CalibratorCalibrator53.70
CalibratorGPT-5.6-Sol52.30
GPT-5.6-SolCalibrator52.10

The full Calibrator performs best, showing improvements in both experience quality and decisions about when to write.

W4: Section 6.4 changes both demonstration sources and output targets in the joint-distillation comparison. The action-only model uses successful trajectories from ordinary GPT-5.6-Sol, while the joint model uses successful trajectories generated with the calibrator. The latter may therefore receive different action demonstrations. Equal data amounts and five training seeds control some factors, but do not separate demonstration quality from generating calibration text.

Response: We add an action-only model trained on the same successful GPT-5.6-Sol + Calibrator demonstrations as the joint model. Both student models are Qwen3-8B and share the same action targets. The action-only model is trained to output the action. The joint model additionally includes a calibration instruction in its prompt and calibration text in its training targets. It generates calibration followed by the action in one response. Both conditions use five training random seeds and the paper’s task lists, with 500 tasks per environment and seed. The table reports mean success rates (%) across the five seeds:

MethodALFWorldAppWorldScienceWorldWebShop
Action-only model84.8020.6050.4031.40
Joint calibration + action model89.9224.3656.6037.96
Gain (pp)+5.12+3.76+6.20+6.56

With the same teacher demonstrations, joint generation improves success in all four environments. Mean success across four environments and five seeds increases from 46.80% to 52.21%.

Sorry for the interruption. As the rebuttal period is approaching its end, we would greatly appreciate it if you could take a moment to review our responses to your comments.

We hope that our clarifications have adequately addressed your concerns. If so, we would be grateful if you could consider updating your rating accordingly. If you have any remaining concerns or questions, please feel free to let us know, and we would be happy to address them.

Reviewer_E8kDOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether language agents use the action-outcome feedback in their within-task interaction history to decide whether an earlier approach should be continued, adjusted, or abandoned, a capability the authors call action calibration. Diagnostic experiments with five actors (GPT-5.6-Sol to Qwen3-8B) in ALFWorld, ScienceWorld, WebShop, and AppWorld show that (i) adding a later segment of a failed trajectory as non-executed reference history lowers recovery from a restored L/3 state (16.60% to 13.20%), and (ii) shuffling historical action strings while keeping observations in order reduces full-rollout success only modestly (46.39% to 43.84%). From these results, the authors hypothesize that ordinary history does not reliably elicit outcome-based reassessment of past actions.

To test this hypothesis, they label each observation as the outcome of its preceding action (Causal Transition), which raises success to 48.25% and lowers next-action repetition on fixed histories, whereas retaining actor-generated experiences lowers success to 45.01%. They then train a Qwen3-8B calibrator with SFT on 10,000 decision contexts: GPT-5.6-Sol proposes candidate experiences, an LLM judge screens them, and targets are assigned by a cross-actor margin between judged next-action improvements and degradations. The frozen calibrator raises success to 51.45% on held-out tasks across all five actors, raises recovery with reference history to 19.9%, and, when integrated into a distilled Qwen3-8B actor, improves success from 47.21% to 52.21%.

Critical strengths
  • All interventions share tasks, restored states, and action budgets across five actors and four environments (10,000 episodes per condition), and the main findings are replicated with native chat messages (Appendix F). Per-cell success counts are reported for every condition and are internally consistent with the aggregates, which makes the empirical claims easy to verify.
  • A frozen 8B calibrator improves every model-level and environment-level aggregate, including the strongest actor GPT-5.6-Sol (+5.06 pp overall), and the ablation shows that effect-based labels add 1.84 pp over content screening alone (49.61% to 51.45%). Selecting experiences by their measured effect on decisions is, in my view, the most transferable contribution of the paper.
Critical weaknesses
  • In the Reference setting, the actor is not told that the source trajectory ultimately failed, and the L/3 to 2L/3 segment need not contain any failure signal, so following it can be a reasonable choice rather than a calibration failure. The drop may also reflect imitation of a stale plan or confusion about whether reference effects occurred in the restored state, neither of which is analyzed.
  • Observations in these environments often restate the executed action (e.g., the ALFWorld observation in Figure 6), so action-outcome pairings remain recoverable after shuffling, and the small Shuffle-All drop does not show that agents ignore pairings. Moreover, a generic history-caution reminder (Check, 48.60%) matches or exceeds Causal Transition (48.25%), and for Qwen3.5-Plus in ALFWorld, Shuffle-All (374/500) exceeds both Raw (327) and Causal (355), suggesting that the gains may stem from generic skepticism or format perturbation rather than outcome attribution.
  • Additionally, it appears possible that this is a performance variation driven by the specific characteristics of the model or environment, rather than a general characteristic of LLM. Currently, results contradicting this trend exist, and it seems highly likely that the trends in analysis results will differ when using better or smaller models, or in more complex and simpler environments.
  • All supervision comes from GPT-5.6-Sol (candidate generation, content screening, and next-action judging), whereas the Experience baseline uses each actor’s own reflections, so the comparison mixes reviewer quality with learned selection. Untrained GPT-5.6-Sol and untrained Qwen3-8B reviewers with the same prompt are needed to show what the learned selection adds.
  • The recovery result (Figure 8) lacks Causal-only and Calibrator-only controls without reference history, so the gains above Raw may reflect the general benefit of these methods rather than better use of the reference. Similarly, the joint actor is trained on trajectories from GPT-5.6-Sol with the calibrator, which succeed on more tasks than the action-only data, so the two variants differ in data distribution.
Questions for authors
  • Please report Experience with an untrained GPT-5.6-Sol reviewer for all five actors (with and without the content screen), and with an untrained Qwen3-8B reviewer using the same prompt. If the calibrator outperforms its own teacher, what explains this?
  • In the Reference setting, is the actor informed that the source trajectory failed?
  • What fraction of continuations replicate reference actions or presuppose reference effects (e.g., inventory changes), and does the recovery drop persist when the reference is explicitly marked as unsuccessful?
  • Given that the Check reminder matches Causal Transition, how do you distinguish outcome attribution from generic history degradation?
  • A direct binding probe, such as asking the model which past action produced a given observation, or a shuffle test restricted to observations that do not echo the action string, would test the hypothesis more directly. How often do observations contain the executed action?
8 comments
AuthorsOpenReview ↗

Response to Reviewer E8kD (Part 1)

Thank you for recognizing the breadth of our evaluation and the value of selecting experiences by their effects on decisions. We address your comments with controlled comparisons.

W1: In the Reference setting, the actor is not told that the source trajectory ultimately failed, and the L/3 to 2L/3 segment need not contain any failure signal, so following it can be a reasonable choice rather than a calibration failure. The drop may also reflect imitation of a stale plan or confusion about whether reference effects occurred in the restored state, neither of which is analyzed.

Response: We do not disclose the source trajectory’s failure because we test whether agents can independently judge which attempts to retain, revise, or abandon. They should retain helpful reference information and discard harmful information. Our results show that they fail to discard the latter.

To check for confusion about reference execution, we use GPT-5.4 to examine the reference, actual history, and continuation reasoning and actions. It flags assumptions that unexecuted reference events have already occurred in the current environment. Re-executed operations do not count as confusion. We examine 50 trajectories per model–environment cell, 200 per model, and 1,000 overall. Each trajectory counts once; entries are confirmed cases:

ModelALFWorldAppWorldScienceWorldWebShopTotal
GPT-5.6-Sol00101
DeepSeek-V4-Pro01001
Qwen3.5-Plus02002
Qwen3-32B530311
Qwen3-8B930214
Overall1491529

We randomly sampled 10 trajectories labeled as confused and 40 labeled as not confused by GPT-5.4. Human judgments agreed with all 50 labels. Confusion is rare, especially among stronger models such as GPT. Among the 29 confirmed cases, 2 succeed with reference and 4 without it. After excluding these cases, 971 sources remain. Their pooled recovery rates are 13.39% with reference and 16.68% without reference. The decline therefore persists.

W2: Observations in these environments often restate the executed action (e.g., the ALFWorld observation in Figure 6), so action-outcome pairings remain recoverable after shuffling, and the small Shuffle-All drop does not show that agents ignore pairings. Moreover, a generic history-caution reminder (Check, 48.60%) matches or exceeds Causal Transition (48.25%), and for Qwen3.5-Plus in ALFWorld, Shuffle-All (374/500) exceeds both Raw (327) and Causal (355), suggesting that the gains may stem from generic skepticism or format perturbation rather than outcome attribution.

Response: Some ALFWorld and ScienceWorld observations repeat the action. WebShop and AppWorld usually return pages or API data without repeating it. To solve this, for the first two environments, we use GPT-5.4 to rewrite action descriptions as state descriptions. For example, “You open the microwave 1. The microwave 1 is open. In it, you see a cup 1.” becomes “You see a cup 1.”

Four actors each receive 100 tasks per environment: 200 per actor and 800 per condition. The table reports Raw and Shuffle-All success rates (%):

ModelALFWorld: Raw (%)ALFWorld: Shuffle-All (%)ScienceWorld: Raw (%)ScienceWorld: Shuffle-All (%)
GPT-5.6-Sol94918173
Qwen3.5-Plus82793427
Qwen3-32B58471713
Qwen3-8B423454

Shuffle-All shows a larger decline than with the original observations, but the decline is still limited.

The similar gains from Causal and Check are precisely our point. Causal adds no review instruction: it labels observations as outcomes of preceding actions, implicitly prompting reassessment. Check requests reassessment explicitly, so similar performance is expected.

We further combine Causal and Check and compare the following four conditions on the same 2,000 tasks (100 per model–environment cell). Actor rows average four environments; Overall averages the five actors:

ActorNRaw (%)Causal (%)Check (%)Causal + Check (%)
GPT-5.6-Sol40074.2575.7576.7576.25
DeepSeek-V4-Pro40065.7568.5067.5068.00
Qwen3.5-Plus40053.7556.0056.0055.75
Qwen3-32B40030.0031.0029.7531.25
Qwen3-8B40017.0019.2518.5018.50
Overall2,00048.1550.1049.7049.95

Overall, both methods improve over Raw. Their combination does not outperform Causal alone. This is consistent with overlapping roles in history checking.

Shuffling past actions may weaken an unsuitable plan’s influence. This may explain Qwen3.5-Plus’s ALFWorld gain. Such gains occur in only a few settings. The remaining results support our claim rather than a generic skepticism or format-perturbation explanation.

AuthorsOpenReview ↗

Response to Reviewer E8kD (Part 2)

W3: Additionally, it appears possible that this is a performance variation driven by the specific characteristics of the model or environment, rather than a general characteristic of LLM. Currently, results contradicting this trend exist, and it seems highly likely that the trends in analysis results will differ when using better or smaller models, or in more complex and simpler environments.

Response: Our experiments cover five actors, from Qwen3-8B to GPT-5.6-Sol, and four environments, with 500 tasks per model–environment cell. The environments cover embodied interaction, scientific experimentation, web interaction, and application calls, with different task formats and difficulty levels.

In the evaluation with 100 tasks per model–environment cell, Raw rates averaged over five actors are 70.00%, 51.60%, 41.00%, and 30.00% in ALFWorld, AppWorld, ScienceWorld, and WebShop, respectively. Together, these model sizes, task types, and difficulty levels indicate that the phenomenon is relatively widespread.

W4: All supervision comes from GPT-5.6-Sol (candidate generation, content screening, and next-action judging), whereas the Experience baseline uses each actor’s own reflections, so the comparison mixes reviewer quality with learned selection. Untrained GPT-5.6-Sol and untrained Qwen3-8B reviewers with the same prompt are needed to show what the learned selection adds.

Response: We add a fixed GPT-5.6-Sol reviewer with and without content screening, and a fixed untrained Qwen3-8B reviewer, using the same review prompt. Actors and task lists stay fixed. Each model–environment cell has 100 tasks. Actor rows average four environments (400 tasks); Overall averages five actors (2,000 tasks).

ActorRaw (%)Experience (actor self-review) (%)GPT-5.6-Sol (%)Untrained Qwen3-8B (%)GPT + content screening (%)Calibrator (%)
GPT-5.6-Sol74.2574.7574.7571.0080.2579.50
DeepSeek-V4-Pro65.7567.5070.5062.0073.7571.75
Qwen3.5-Plus53.7550.7556.2548.5060.0057.50
Qwen3-32B30.0026.5033.7522.0036.2535.50
Qwen3-8B17.0014.0021.0014.0024.7524.25
Overall48.1546.7051.2543.5055.0053.70

The trained Calibrator outperforms both untrained reviewers: 53.70% versus 51.25% for GPT-5.6-Sol and 43.50% for Qwen3-8B. This reflects the added benefit of learning experience generation and selection. GPT-5.6-Sol with screening performs better still, as expected because the calibrator is trained to fit it.

W5: The recovery result (Figure 8) lacks Causal-only and Calibrator-only controls without reference history, so the gains above Raw may reflect the general benefit of these methods rather than better use of the reference. Similarly, the joint actor is trained on trajectories from GPT-5.6-Sol with the calibrator, which succeed on more tasks than the action-only data, so the two variants differ in data distribution.

Response: We add Causal, Experience, and Calibrator controls without reference. All conditions use the same 1,000 sources. Rates average five actors and four environments. Each continuation restores L/3 and allows at most 30 new actions. Reference conditions add the L/3–2L/3 segment and retain each method’s original history-processing scheme.

MethodWithout reference (%)With reference (%)
Raw16.6013.20
Causal18.4018.10
Experience16.1017.80
Calibrator19.2019.90

With reference, Causal and Calibrator outperform Raw by 4.90 and 6.70 pp, respectively. These gains show that better feedback organization and calibration improve recovery.

Ordinary reference reduces Raw success by 3.40 pp. Causal performs similarly with and without it, showing that it better uses reference information while discarding harmful information. Reference improves Experience and Calibrator by 1.70 and 0.70 pp, respectively. Both therefore gain more than they lose from the reference.

Joint training. We add an action-only Qwen3-8B baseline using the same successful GPT-5.6-Sol + Calibrator trajectories and action targets as the joint model. The joint model additionally includes a calibration instruction in its prompt and calibration text in its training targets. It generates calibration followed by the action in one response. Both variants use five training random seeds and the paper’s task lists, with 500 tasks per environment and seed. Mean success across four environments and five seeds rises from 46.80% to 52.21%, a gain of 5.41 pp from joint generation.

MethodALFWorldAppWorldScienceWorldWebShop
Action-only model84.8020.6050.4031.40
Joint calibration + action model89.9224.3656.6037.96
Gain (pp)+5.12+3.76+6.20+6.56
AuthorsOpenReview ↗

Response to Reviewer E8kD (Part 3)

Q1: Please report Experience with an untrained GPT-5.6-Sol reviewer for all five actors (with and without the content screen), and with an untrained Qwen3-8B reviewer using the same prompt. If the calibrator outperforms its own teacher, what explains this?

Response: Please see our response to W4.

Q2: In the Reference setting, is the actor informed that the source trajectory failed?

Response: No. In the default setting, the model must assess the value of the reference segment from the actions and feedback it contains. Disclosing the eventual failure of the source trajectory would provide extra information that influences this judgment.

Q3: What fraction of continuations replicate reference actions or presuppose reference effects (e.g., inventory changes), and does the recovery drop persist when the reference is explicitly marked as unsuccessful?

Response: Reference actions include both useful and unhelpful attempts. Counting action reuse alone therefore cannot distinguish appropriate use of history from following unsuitable attempts. Using the procedure described in W1, we examine whether the model incorrectly assumes that events in the reference have already occurred in the current environment. We confirm such confusion in 29 of 1,000 Reference trajectories (2.90%); the breakdown is given in W1 above.

We add experiments in which we explicitly mark the reference as unsuccessful and compare the following three conditions on the same 1,000 source trajectories. The failure-notice condition adds a factual statement that the source trajectory ultimately failed. All other reference content, the restored state, and the continuation budget remain unchanged. Success counts sum all model–environment cells; rates use all 1,000 sources.

ConditionSuccessesSuccess rate (%)
Raw, without reference16616.60
Ordinary Reference13213.20
Reference explicitly marked as failed15115.10

The failure notice increases recovery from 13.20% to 15.10%, which remains below Raw without reference (16.60%). This shows that even when the reference is explicitly marked as unsuccessful, the actor still fails to take advantage of the information.

Q4: Given that the Check reminder matches Causal Transition, how do you distinguish outcome attribution from generic history degradation?

Response: The similar performance of Causal and Check is precisely the finding we intend to highlight. Causal makes action outcomes clear to implicitly remind the actor to reassess earlier attempts. Check makes this reminder explicit, so similar performance is expected. Their controlled comparison on the same tasks is provided in W2 above.

Q5: A direct binding probe, such as asking the model which past action produced a given observation, or a shuffle test restricted to observations that do not echo the action string, would test the hypothesis more directly. How often do observations contain the executed action?

Response: Please see our response to W2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.