- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces Counterfactual Policy Learning (CPL), a four-stage process for deciding which decisions from experience deserve persistent memory. It first contrasts successful and failed trajectories from similar tasks, abstracts candidate decision differences into policies with executable alternatives, validates each candidate by replacing the decision in a replayable same-task trajectory, and admits only policies with positive intervention effects and sufficient confidence. The paper evaluates candidate quality, intervention effects, policy reuse, and end-task performance on LoCoMo, ALFWorld, and StrategyQA. The reported improvements are limited and inconsistent: the executable ALFWorld evidence is based on a small evaluation, while the reasoning-only LoCoMo results do not provide strong causal or generalization evidence.
Critical strengths
- The paper studies an important and underexplored question: among the many decisions contained in successful and failed agent trajectories, which ones are sufficiently reliable to retain as reusable skills or memory? This is consequential because indiscriminate experience storage can propagate incidental behavior and negative transfer.
- The proposed conceptual separation between observational candidate discovery, intervention-based validation, and later-task reuse is sensible. In particular, the paper correctly recognizes that a successful-versus-failed trajectory contrast alone is not causal evidence.
Critical weaknesses
- The core procedure is not specified well enough to reproduce or interpret reliably. , , and are LLM-generated scores without operational formulas, calibrated rubrics, or sufficiently detailed prompts; the main value of is not reported. The admission weights are fixed to without justification or sensitivity analysis, while the sweep uses only ten candidate policies. More seriously, the formal construction changes a policy-consistent action into a policy-violating alternative, whereas the ALFWorld protocol starts from a failed trajectory and replaces an incorrect action with a policy-guided action. The paper does not reconcile these opposite intervention directions with its definition of .
- The empirical evidence is too weak to support the proposed multi-stage framework. The paper does not clearly report how many successful and failed source trajectories are required. Only 4 of 389 intervention-testable contrastive candidates have positive , compared with 8 of 512 random candidates (reported ), so the candidate mechanism is not shown to improve intervention validity. The ALFWorld end-task evaluation contains only 26 tasks, and the full CPL configuration admits 0/17 policies in one reported ablation, causing the reuse path to collapse to the base agent. LoCoMo interventions are reasoning-level perturbations rather than executable environmental interventions. These issues substantially weaken both the effectiveness and causal interpretation claims.
Questions for authors
- Which intervention direction is actually used to compute the entering the admission rule: policy-consistent to policy-violating, as defined in Section 3.3 and the prompt, or failed/incorrect to policy-guided, as described for ALFWorld? Please give a single consistent definition and recompute the relevant results if necessary.
- What are the exact algorithms or scoring rubrics for , , and , and what value of is used in each main experiment? How were the four admission weights selected, and how sensitive are results to them?
- How many successful trajectories, failed trajectories, held-out validation trajectories, and environment executions are required to construct the reported policy memory? Please report results across multiple independently constructed experience pools and random seeds.
2 comments
The authors’ response does not resolve my core concerns regarding the reproducibility, methodological consistency, and empirical support of CPL. Instead, it confirms that several central scores lack reproducible definitions, key hyperparameters are insufficiently justified, and the reported admission analysis is inconsistent with the method’s own hard-gating rule. The submitted PDF has not been updated to address these issues. Therefore, I maintain my original rating.
Response to Reviewer 5ENa
Thank you for these questions. We clarify the intervention direction, scoring procedure, and scope of the reported repetitions.
1. Intervention direction used for admission.
Section 3.3 and the intervention prompt describe break-direction interventions: replacing a policy-consistent decision with a policy-violating alternative. The ALFWorld correction experiment and Table 6 instead use fix-direction interventions: replacing an incorrect decision with a policy-guided alternative.
A common sign convention is:
\left[Perf(\tau_i^{consistent})-Perf(\tau_i^{comparison})\right].$$ For break-direction interventions, the comparison is the policy-violated continuation. For fix-direction interventions, it is the original failed continuation. Positive values favor the policy-consistent decision in both cases. However, the two directions evaluate different starting states and contrasts; their numerical estimates are not interchangeable. **Table 6 explicitly uses fix-direction IE in the confidence score.** Its seven-policy count at threshold 0.5 is inconsistent with the additional hard gate: $$IE(P)>0 \land Conf_{adm}(P)\geq\theta_{adm}.$$ Only `P_USE_TOOL_look_at_obj_in_light` has positive fix-direction IE, reported as 0.20. Applying both conditions to the reported estimates leaves only this policy eligible at threshold 0.5. Thus, the seven-policy row cannot represent the complete hard-gated admission rule. Its reuse and survival scores cannot be transferred to the one-policy configuration without rerunning that evaluation. The available aggregate results do not support a numerical recomputation of those downstream scores. **2. Scoring rubrics, relevance thresholds, and weights.** The reported scores rely on structured LLM assessments: - **Rel(P):** Candidate relevance derived from state–action alignment, recurrence across experiences, and LLM judgment. This is an observational relevance signal. - **Cons(P):** Consistency of the recommended decision direction across supporting experiences. - **Evidence(P):** Amount of supporting evidence and its agreement with the induced policy. The assessment prompt requests Cons(P) and Evidence(P) in [0, 1], but does not specify a numerical rubric mapping evidence counts or observed agreement to particular scores. The submission therefore describes their intended meanings without fully specifying reproducible estimators. The admission score uses fixed weights: $$Conf_{adm}(P)= 0.25\,Rel(P)+0.40\,\widehat{IE}(P) +0.20\,Cons(P)+0.15\,Evidence(P).$$ Intervention evidence receives the largest coefficient. No empirical weight-selection procedure or sensitivity analysis over these four weights is reported. Table 6 varies the admission threshold while holding the weights fixed; it does not demonstrate robustness to alternative weight choices. A relevance threshold of 0.6 defines the high-relevance subset in the candidate-quality analysis and the candidate pool for the ten-policy threshold experiment. The submission does not list the relevance threshold separately for every main experiment. Consequently, a complete experiment-by-experiment threshold specification is missing. **3. Experience requirements and independent repetitions.** The available records provide these counts: | Experiment | Reported size | |---|---| | StrategyQA construction | 600 contrastive pairs, 1,630 candidate decisions, 17 induced policies | | StrategyQA validation | 12 validation tasks per policy configured; 392 recorded interventions | | StrategyQA unseen evaluation | 687 questions; full CPL admits 0/17 policies | | L2 candidate analysis | 534 candidates; 389 intervention-testable | | Threshold analysis | 10 policies induced from 145 high-relevance candidates | | ALFWorld correction | 74 intervention points per arm; 8 policy-guided recoveries versus 0 random recoveries | These are not counts of unique successful, failed, and validation trajectories. Contrastive pairs may reuse trajectories, and intervention records need not correspond to distinct episodes. The reported ALFWorld intervention counts also do not account separately for executions used to collect failures, construct candidates, replay prefixes, or evaluate continuations. They therefore do not provide the total environment-execution cost of constructing memory. Separate ALFWorld end-task records contain three seeds, each yielding 1/26 successful CPL episodes, or 3/78 overall. These are end-task repetitions, not repetitions over independently reconstructed experience pools and policy memories. No repeated memory-construction experiment across independent pools and seeds is reported. The current results characterize the particular construction runs evaluated; they do not establish the stability or expected construction cost of a non-empty, intervention-validated memory.