acceptodds
Under review as a conference paper at ICLR 2027

Learning What Matters: Counterfactual Policy Learning for LLM Agents

Abstract

Large language model (LLM) agents increasingly improve through accumulated interaction experience, yet it remains unclear which decisions within a trajectory are worth learning and reusing. Successful trajectories may contain many incidental actions, so trajectory-level memorization cannot reliably distinguish essential decisions from irrelevant details. We propose Counterfactual Policy Learning (CPL), a decision-level experience learning framework that separates candidate identification from policy validation. CPL contrasts successful and failed trajectories to identify candidate decision differences, treating this observational signal only as relevance rather than causal evidence. It then abstracts candidates into intervention-testable policies and performs same-task interventions that replace policy-consistent decisions with alternative actions. Only policies with positive intervention effects and sufficient admission confidence are admitted into policy memory; unseen-task performance is used solely for downstream evaluation. On ALFWorld, policy-guided interventions recover 10.8% of failed episodes versus 0% for random replacements; on LoCoMo, CPL achieves 32.65% overall Semantic Similarity. Ablations show that type-matched retrieval acts as a firewall against negative transfer, reducing ReusePerf from 53.86% to 43.81% when replaced by random retrieval. CPL thus reframes experience learning from “what happened?” to “which decision is worth learning from?”Our code is available at https://github.com/linda-2-hh/Counterfactual-Policy-Learning.

Primary area:
interpretability and explainable AI
Keywords:
Counterfactual Policy Learning, LLM Agents, Agent Memory, Continual Learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_5ENaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Counterfactual Policy Learning (CPL), a four-stage process for deciding which decisions from experience deserve persistent memory. It first contrasts successful and failed trajectories from similar tasks, abstracts candidate decision differences into policies with executable alternatives, validates each candidate by replacing the decision in a replayable same-task trajectory, and admits only policies with positive intervention effects and sufficient confidence. The paper evaluates candidate quality, intervention effects, policy reuse, and end-task performance on LoCoMo, ALFWorld, and StrategyQA. The reported improvements are limited and inconsistent: the executable ALFWorld evidence is based on a small evaluation, while the reasoning-only LoCoMo results do not provide strong causal or generalization evidence.

Critical strengths
  1. The paper studies an important and underexplored question: among the many decisions contained in successful and failed agent trajectories, which ones are sufficiently reliable to retain as reusable skills or memory? This is consequential because indiscriminate experience storage can propagate incidental behavior and negative transfer.
  2. The proposed conceptual separation between observational candidate discovery, intervention-based validation, and later-task reuse is sensible. In particular, the paper correctly recognizes that a successful-versus-failed trajectory contrast alone is not causal evidence.
Critical weaknesses
  1. The core procedure is not specified well enough to reproduce or interpret reliably. Rel(P)\mathrm{Rel}(P), Cons(P)\mathrm{Cons}(P), and Evidence(P)\mathrm{Evidence}(P) are LLM-generated scores without operational formulas, calibrated rubrics, or sufficiently detailed prompts; the main value of θrel\theta_{\mathrm{rel}} is not reported. The admission weights are fixed to (0.25,0.40,0.20,0.15)(0.25, 0.40, 0.20, 0.15) without justification or sensitivity analysis, while the θadm\theta_{\mathrm{adm}} sweep uses only ten candidate policies. More seriously, the formal construction changes a policy-consistent action into a policy-violating alternative, whereas the ALFWorld protocol starts from a failed trajectory and replaces an incorrect action with a policy-guided action. The paper does not reconcile these opposite intervention directions with its definition of IE(P)IE(P).
  2. The empirical evidence is too weak to support the proposed multi-stage framework. The paper does not clearly report how many successful and failed source trajectories are required. Only 4 of 389 intervention-testable contrastive candidates have positive IEIE, compared with 8 of 512 random candidates (reported p=0.57p=0.57), so the candidate mechanism is not shown to improve intervention validity. The ALFWorld end-task evaluation contains only 26 tasks, and the full CPL configuration admits 0/17 policies in one reported ablation, causing the reuse path to collapse to the base agent. LoCoMo interventions are reasoning-level perturbations rather than executable environmental interventions. These issues substantially weaken both the effectiveness and causal interpretation claims.
Questions for authors
  1. Which intervention direction is actually used to compute the IE(P)IE(P) entering the admission rule: policy-consistent to policy-violating, as defined in Section 3.3 and the prompt, or failed/incorrect to policy-guided, as described for ALFWorld? Please give a single consistent definition and recompute the relevant results if necessary.
  2. What are the exact algorithms or scoring rubrics for Rel(P)\mathrm{Rel}(P), Cons(P)\mathrm{Cons}(P), and Evidence(P)\mathrm{Evidence}(P), and what value of θrel\theta_{\mathrm{rel}} is used in each main experiment? How were the four admission weights selected, and how sensitive are results to them?
  3. How many successful trajectories, failed trajectories, held-out validation trajectories, and environment executions are required to construct the reported policy memory? Please report results across multiple independently constructed experience pools and random seeds.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 5ENa

Thank you for these questions. We clarify the intervention direction, scoring procedure, and scope of the reported repetitions.

1. Intervention direction used for admission.

Section 3.3 and the intervention prompt describe break-direction interventions: replacing a policy-consistent decision with a policy-violating alternative. The ALFWorld correction experiment and Table 6 instead use fix-direction interventions: replacing an incorrect decision with a policy-guided alternative.

A common sign convention is:

\left[Perf(\tau_i^{consistent})-Perf(\tau_i^{comparison})\right].$$ For break-direction interventions, the comparison is the policy-violated continuation. For fix-direction interventions, it is the original failed continuation. Positive values favor the policy-consistent decision in both cases. However, the two directions evaluate different starting states and contrasts; their numerical estimates are not interchangeable. **Table 6 explicitly uses fix-direction IE in the confidence score.** Its seven-policy count at threshold 0.5 is inconsistent with the additional hard gate: $$IE(P)>0 \land Conf_{adm}(P)\geq\theta_{adm}.$$ Only `P_USE_TOOL_look_at_obj_in_light` has positive fix-direction IE, reported as 0.20. Applying both conditions to the reported estimates leaves only this policy eligible at threshold 0.5. Thus, the seven-policy row cannot represent the complete hard-gated admission rule. Its reuse and survival scores cannot be transferred to the one-policy configuration without rerunning that evaluation. The available aggregate results do not support a numerical recomputation of those downstream scores. **2. Scoring rubrics, relevance thresholds, and weights.** The reported scores rely on structured LLM assessments: - **Rel(P):** Candidate relevance derived from state–action alignment, recurrence across experiences, and LLM judgment. This is an observational relevance signal. - **Cons(P):** Consistency of the recommended decision direction across supporting experiences. - **Evidence(P):** Amount of supporting evidence and its agreement with the induced policy. The assessment prompt requests Cons(P) and Evidence(P) in [0, 1], but does not specify a numerical rubric mapping evidence counts or observed agreement to particular scores. The submission therefore describes their intended meanings without fully specifying reproducible estimators. The admission score uses fixed weights: $$Conf_{adm}(P)= 0.25\,Rel(P)+0.40\,\widehat{IE}(P) +0.20\,Cons(P)+0.15\,Evidence(P).$$ Intervention evidence receives the largest coefficient. No empirical weight-selection procedure or sensitivity analysis over these four weights is reported. Table 6 varies the admission threshold while holding the weights fixed; it does not demonstrate robustness to alternative weight choices. A relevance threshold of 0.6 defines the high-relevance subset in the candidate-quality analysis and the candidate pool for the ten-policy threshold experiment. The submission does not list the relevance threshold separately for every main experiment. Consequently, a complete experiment-by-experiment threshold specification is missing. **3. Experience requirements and independent repetitions.** The available records provide these counts: | Experiment | Reported size | |---|---| | StrategyQA construction | 600 contrastive pairs, 1,630 candidate decisions, 17 induced policies | | StrategyQA validation | 12 validation tasks per policy configured; 392 recorded interventions | | StrategyQA unseen evaluation | 687 questions; full CPL admits 0/17 policies | | L2 candidate analysis | 534 candidates; 389 intervention-testable | | Threshold analysis | 10 policies induced from 145 high-relevance candidates | | ALFWorld correction | 74 intervention points per arm; 8 policy-guided recoveries versus 0 random recoveries | These are not counts of unique successful, failed, and validation trajectories. Contrastive pairs may reuse trajectories, and intervention records need not correspond to distinct episodes. The reported ALFWorld intervention counts also do not account separately for executions used to collect failures, construct candidates, replay prefixes, or evaluate continuations. They therefore do not provide the total environment-execution cost of constructing memory. Separate ALFWorld end-task records contain three seeds, each yielding 1/26 successful CPL episodes, or 3/78 overall. These are end-task repetitions, not repetitions over independently reconstructed experience pools and policy memories. No repeated memory-construction experiment across independent pools and seeds is reported. The current results characterize the particular construction runs evaluated; they do not establish the stability or expected construction cost of a non-empty, intervention-validated memory.

The authors’ response does not resolve my core concerns regarding the reproducibility, methodological consistency, and empirical support of CPL. Instead, it confirms that several central scores lack reproducible definitions, key hyperparameters are insufficiently justified, and the reported admission analysis is inconsistent with the method’s own hard-gating rule. The submitted PDF has not been updated to address these issues. Therefore, I maintain my original rating.

Reviewer_NVHUOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Counterfactual Policy Learning (CPL), which extracts candidate policies from successful–failed trajectory contrasts and validates them through interventions before storing them in memory. Its main contribution is to separate observational relevance, intervention evidence, and cross-task reuse. Experiments examine task performance and policy admission, with mixed evidence for effective policy reuse.

Critical strengths
  1. The key research question of determining which decision-level experience is worth remembering is meaningful, so this paper targets an important problem.

  2. Using interventions as a policy-admission criterion is a useful idea for reducing unsupported memories. The explicit distinction between observational relevance and intervention evidence provides a reasonable framework for studying what agents should retain.

  3. The intervention experiment provides preliminary behavioral evidence: policy-guided fixes recover partial failed cases, compared with zero for random valid replacements. The paper also acknowledges negative findings rather than presenting all interventions as successful.

Critical weaknesses
  1. While the idea of counterfactual policy learning is reasonable, the core technical contributions of this work are somewhat limited. Particularly, the proposed contrastive decision identification and policy admission lack clear motivations why these strategies are new and how they work well.

  2. The experiments do not convincingly establish the central contribution of learning useful, reusable policies. For example, L2 selection also does not improve positive intervention rates over random selection. End-task evidence is limited: CPL solves only very few ALFWorld tasks, below A-Mem, and its LoCoMo advantage over RAG-BM25 is small without uncertainty estimates.

  3. Inconsistent definitions and reporting undermine soundness and readability. For example, Table 6 admits seven policies at threshold 0.5 despite the text stating that only one candidate has positive fix-direction IE, conflicting with the mandatory positive-IE gate. Numerous incomplete sentences in Sections 4.2–4.4 further obscure the experimental interpretation.

Questions for authors
  1. What exactly does ReusePerf measure in each table?

  2. Was the hard (IE(P)>0) gate applied in Table 6? How can seven policies pass when only one has positive fix-direction IE? Please provide the per-policy intervention effects and admission scores.

  3. Which experiment demonstrates that a non-empty, intervention-validated memory improves unseen-task performance over the same agent without memory? Please connect the admitted policies and retrieval coverage to the reported LoCoMo result.

  4. For the ALFWorld interventions, were policy fixes derived from expert plans, and how do they compare with an LLM-generated correction without CPL validation?

  5. Please specify the backbone, data splits, and Semantic Similarity implementation for the main LoCoMo comparison, together with uncertainty estimates. Where are the StrategyQA results announced in Section 4.1?

2 comments
AuthorsOpenReview ↗

Response to Reviewer NVHU

Thank you for these questions. We clarify the metrics and distinguish evidence for preventing negative transfer from evidence for beneficial reuse.

1. What ReusePerf measures.

The manuscript uses this label for different quantities, which is a reporting error.

  • Table 4: Answer accuracy over all 687 held-out StrategyQA questions, including those receiving no policy. Full CPL’s 57.21% is therefore base-agent accuracy with empty memory. Without L1, accuracy is 53.86%, with retrieval on 186/687 questions (27.07%). On this covered subset, accuracy is 46.77%, versus 59.14% without policy injection.
  • Table 6: Performance at policy firings on the test split; Survival measures completed test episodes. A 100% firing-level score does not establish 100% episode success or improvement over the agent without memory.
  • Table 14: “–” denotes undefined retrieval-conditioned reuse when no policy is admitted or retrieved.

These quantities differ from the paired improvement fraction in Section 3.4. Appendix B.3 also counts outcome changes rather than improvements. These definitions should not have been used interchangeably.

2. The hard gate and Table 6.

Seven admitted policies at threshold 0.5 cannot satisfy the stated rule when only one candidate has positive fix-direction IE:

IE(P)>0∧Confadm(P)≥θadmIE(P)>0 \land Conf_{adm}(P)\geq\theta_{adm}

Table 6 therefore cannot substantiate sensitivity of the complete admission rule. Its counts reflect confidence-threshold retention without establishing satisfaction of the separate positive-IE gate.

The individually reported policy is P_USE_TOOL_look_at_obj_in_light, with fix-direction IE of 0.20. Retention at 0.6 and rejection at 0.7 place its confidence score in the interval [0.6, 0.7). The other nine candidates have non-positive fix-direction IE and fail the hard gate regardless of confidence. Applying the stated rule to these reported estimates leaves only this policy eligible at threshold 0.5.

The aggregate sweep does not provide the remaining individual IE values or exact confidence scores; those cannot be reconstructed from retention counts. Break-direction saturation does not reconcile the discrepancy because the sweep explicitly specifies fix-direction IE.

3. Evidence for beneficial unseen-task reuse.

No reported experiment establishes the complete chain requested here. In the 687-question StrategyQA experiment, full CPL admits 0/17 policies and has zero retrieval coverage. Its 57.21% accuracy demonstrates fallback to the base agent, not improvement from validated memory. Removing L1 admits six policies and lowers accuracy to 53.86%; random retrieval with that same memory lowers it further to 43.81%. These results support filtering harmful candidates and limiting their exposure.

The separate LoCoMo comparison reports 32.65% Semantic Similarity, versus 31.81% for RAG-BM25 and 16.71% for LLM-Only. However, it does not connect admitted-policy identities, validation records, and retrieval coverage to individual predictions. It therefore does not isolate the contribution of intervention-validated memory. Same-task ALFWorld recovery likewise demonstrates actionability at tested states, rather than unseen-task transfer.

4. Expert plans and the correction baseline.

Expert plans serve as successful references in the documented ALFWorld construction. Appendix A.1 explicitly contrasts a failed trajectory with a same-task expert plan and derives corrective actions from that contrast. This is expert-assisted correction.

The reported comparison is policy-guided correction versus random valid replacement: 8/74 recoveries versus 0/74. No matched LLM-generated correction baseline without CPL validation is reported. The experiment therefore shows that the constructed corrections can recover failures, but does not separate the contribution of expert guidance from that of CPL validation.

5. LoCoMo configuration, uncertainty, and StrategyQA.

The experiment summary records 1,540 LoCoMo QA pairs. However, the submission does not specify the exact Table 1 backbone/checkpoint, conversation-level split membership, or Semantic Similarity encoder, preprocessing, and scoring implementation. DeepSeek-7B is identified for ALFWorld and the StrategyQA admission experiment; this does not establish Table 1’s configuration.

Table 1 provides point estimates without seed variability or confidence intervals. The 0.84-percentage-point difference from RAG-BM25 is descriptive; statistical significance is not established.

The 687-question admission ablation is the StrategyQA experiment. Full CPL obtains 57.21% EM, equal to the base agent because no policy is admitted. The standalone StrategyQA comparison announced in Section 4.1 is absent from the submitted results. This ablation supports conservative admission under the tested conditions, not a positive StrategyQA accuracy gain.

Thanks for the detailed responses. After reading the rebuttal, my concern about the technical novelty of this work still remains, while I appreciate the additional clarifications. The paper in its current presentation form, is still difficult to read.

Reviewer_aYfuOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Counterfactual Policy Learning (CPL) is a decision-level framework that enables LLM agents to reuse past experiences by extracting actionable policies from successful and failed trajectories. Rather than memorizing entire trajectories, CPL identifies meaningful decision differences that can guide future actions.

CPL operates in four stages: L2 Contrast aligns successful and failed trajectories to identify relevant decision differences; Policy Induction transforms these differences into structured policies containing context, goals, strategies, actions, restrictions, and evidence; L1 Intervention evaluates policies through counterfactual action replacements; and Admission stores only policies with positive intervention effects and sufficient confidence. Generalization to unseen tasks (L3) is evaluated separately.

Experimental results demonstrate CPL's effectiveness. In ALFWorld, policy-guided interventions recovered 10.8% of failed episodes (8/74), compared with 0% for random replacements. On LoCoMo, CPL achieved 32.65% overall Semantic Similarity. An ablation across 687 held-out instances showed that type-matched retrieval improved ReusePerf from 43.81% to 53.86%, highlighting the importance of retrieving relevant policies.

Critical strengths

CPL introduces a principled framework that separates observational relevance (L2), interventional validation (L1), and generalization to unseen tasks (L3). Requiring interventional support before admitting policies into memory is a promising direction for agent experience reuse. The paper also reports negative findings transparently, including ineffective LoCoMo interventions, no significant advantage of L2 candidates over random selection (Fisher p = 0.57), and the small ALFWorld evaluation sample.

Critical weaknesses

The main weakness is the lack of evidence that admitted policies improve downstream performance. Full CPL admits no policies in the 687-instance ablation, leaving its central reuse claim unvalidated. Inconsistent ReusePerf definitions and potentially incomparable baseline values further prevent reliable interpretation. The admission gate is also unclear, particularly regarding break-direction versus fix-direction intervention effects, while threshold-selection statements appear contradictory. End-task results are limited, with CPL trailing stronger baselines on ALFWorld and LongMemEval. Additionally, truncated text, omitted stronger baselines in figures, contradictory appendix claims, and unspecified models and datasets undermine reporting clarity and reproducibility.

Questions for authors
  1. Can the authors demonstrate that admitted policies improve held-out performance over the base agent?
  2. How is ReusePerf defined across Tables 4, 6, and 14, especially when no policies are admitted, and what does the 57.21 baseline represent?
  3. Does the admission gate use break-direction or fix-direction intervention effects, and how are the reported admission counts reconciled?
  4. Which LLMs are used for relevance scoring, policy induction, and intervention construction? Which model and benchmark are used for LoCoMo and the 687-instance ablation, respectively?
  5. How were the 74 ALFWorld failure cases and intervention points selected, and were policy actions derived from same-task expert trajectories?
1 comment
AuthorsOpenReview ↗

Response to Reviewer aYfu

Thank you for these questions. Our code audit revealed unclear metric definitions and benchmark-specific procedures. We clarify them below.

1. Do admitted policies improve held-out performance?

We cannot yet make this claim for the full CPL configuration. In StrategyQA v2, 17 policies were induced and evaluated through 392 interventions, but none passed admission: six had non-positive IE, and eleven had confidence below 0.40 (mean 0.2837; maximum 0.337). Thus, no policy was used on the held-out set.

The no-L1 ablation illustrates the gate's role, not full-CPL gains. Admitting six policies reduced held-out accuracy from 57.21% to 53.86%. On the 186 covered questions, accuracy fell from 59.14% to 46.77%; uncovered questions were unchanged at 56.49%. Three admitted policies also had negative IE. ALFWorld repairs recovered 8/74 cases versus 0/74 for random valid replacements; this is repair-level, not transfer, evidence.

We will narrow our claim and test positive transfer using independent experience pools with non-empty admitted memories and separate held-out tasks across multiple seeds. Until then, we will not claim positive transfer.

2. What does ReusePerf mean, and what is 57.21?

The original tables conflate ReusePerf with overall accuracy. ReusePerf is the fraction of tasks on which a retrieved policy improves the outcome over the same base agent:

\frac{N_{\mathrm{improved}}(P)} {N_{\mathrm{retrieved}}(P)}.$$ When no policy is admitted or retrieved, ReusePerf is undefined (N/A), rather than being assigned the base agent's accuracy. The 57.21% figure is the base agent's accuracy on the 687 held-out StrategyQA questions, not ReusePerf. The 53.86% figure is the no-L1 overall held-out accuracy. If no policy is admitted or retrieved, ReusePerf is N/A; the system falls back to the base agent. We will revise Tables 4, 6, and 14 to report held-out accuracy, policy coverage, and ReusePerf separately. **3. Which intervention direction is used, and how do the admission counts fit together?** Admission uses **break-direction** intervention: from a policy-consistent trajectory, we replace the supported action with a policy-violating alternative and compute $$IE(P)=\mathbb E_\tau[\mathrm{Perf}(\tau)-\mathrm{Perf}(\tau_{\mathrm{violated}})].$$ Positive IE means policy violation lowers performance. The **fix-direction** ALFWorld procedure starts from a failed trajectory and replaces an incorrect action with a policy-guided one. This is a separate repair evaluation, not the admission IE. In StrategyQA v2, 0/17 policies were admitted: six were rejected for $IE\leq0$, and eleven failed the 0.40 confidence threshold. The earlier ten-policy threshold sweep used a separate candidate pool and should not be compared with this result. We will remove it from the main comparison or report it separately after verification. These are benchmark-specific: ALFWorld uses an IE-based gate (threshold 0.50); LoCoMo has no environmental IE term (threshold 0.50); StrategyQA v2 uses threshold 0.40. We will make this distinction explicit. **4. Which models are used?** In ALFWorld, Rel and policy induction share one DeepSeek-Chat-7B instance (`deepseek-chat-pytorch-7b`, FP32). Intervention construction is programmatic: actions are replaced and trajectories replayed in the real environment, without an LLM call. Admission is numerical. In LoCoMo, each run uses one shared backbone for graph construction, candidate extraction, Rel/Cons scoring, and QA. Backbones are Llama-2-Chat-7B, DeepSeek-Chat-7B, and GLM4-Chat-9B. `all-MiniLM-L6-v2` is used for embeddings and retrieval, not as the LLM backbone. The 687-instance ablation is on StrategyQA with DeepSeek-7B. The available DeepSeek LoCoMo record is an earlier raw-concatenation run, not yet verified against the 1,540-instance Llama-2 and GLM4 pipeline. We will rerun it or qualify it before comparison. **5. How were the 74 ALFWorld cases selected, and were replacements from same-task expert trajectories?** The records show 74 intervention points from failed trajectories: policy-guided replacements recovered 8 cases, versus 0 for random valid replacements. However, current records do not establish whether every replacement came directly from a same-task expert trajectory. We will trace each intervention to its episode, step, policy, source experience, and replacement action, and document the selection procedure. We will also compare same-task successful/expert actions where available, random valid actions, and unconstrained LLM-generated corrections without CPL validation. Thank you again for identifying these gaps. We will correct the paper and report proposed experiments as future validation, not as completed results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.