acceptodds
Withdrawn from ICLR 2027

Policy Improvement as Decision Revision: Preserving the Incumbent under Executable Feedback

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Executable counterfactual evaluation provides evidence about alternative actions, but utilities alone do not specify the decision that this evidence is meant to revise. We formalize the evaluation object as an evaluated decision, (p_old, Q), pairing the incumbent action distribution with executable utilities over a common semantic-action support. Projecting this object onto Q is many-to-one: any supervision constructed solely from Q is invariant to the incumbent and therefore discards its identity before policy improvement begins. This motivates a representation principle for learning from executable feedback: revise the evaluated decision relative to its incumbent rather than replace it with a decision constructed from utilities alone. Distributional Decision Revision (DiRe) instantiates this principle using an established KL-regularized improvement target and an incumbent-relative residual policy parameterization. We test this principle through controlled interventions that separately isolate revision specification and finite-model realization. With executable utilities fixed, changing the incumbent leaves Q-only targets unchanged but changes the DiRe revision. On matched evaluated decisions, Hard-Q, utility-only supervision, and DiRe encode similar expected executable improvements (+0.0645, +0.0625, and +0.0638) while inducing substantially different incumbent-relative displacements (JS 0.1101, 0.1554, and 0.0688), showing that executable gain alone does not characterize the revision. We then hold the DiRe target, model family, cross-entropy objective, optimization budget, and initialization fixed and intervene only on the realization reference. On 59 state-held-out decisions, using the evaluated incumbent rather than a uniform reference reduces target-policy JS from 0.1313 to 0.0572 and changes realized executable improvement from -0.0162 to +0.0053. Exact reconstruction and richer-scorer analyses further separate specification from realization: the target is representable in principle, while a semantic residual scorer realizes 50.9% of its intended improvement on a later state-disjoint batch. Together, these results identify incumbent identity as information that can be lost at the evaluation-to-supervision interface, while finite-model realization remains a distinct and heterogeneous bottleneck. Independent conversation-level generalization remains untested.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Long-Horizon Agents, Agent Memory, Decision-Centric Learning, Trajectory Compression, Experience Selection, Temporal Reasoning, Tool-Using Agents, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qG7QOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies policy improvement from Q-value estimates obtained by rolling out each action under a fixed rollout policy, and argues that the improvement target should depend on the old policy, since a target built from Q-values alone is identical for any two old policies with the same Q-values. It uses the standard KL-regularized target and fits it with a residual parameterization, so the network learns only a log-ratio correction to the old policy instead of a new policy from scratch. In small-scale experiments the residual parameterization moves expected Q in the right direction more reliably than fitting a softmax policy from scratch.

Critical strengths
  1. Separating target construction from target fitting, and varying only the reference distribution while holding everything else fixed, makes it clear that the bottleneck is fitting, not the target.
Critical weaknesses
  1. Hard to read. Many coined terms replace standard RL vocabulary, obscuring the actual idea. And many places of clear ai generated content. (example: "Primary comparisons share the corrected V2 protocol" in Experiment section)
  2. The paper introduces no new method, as the authors acknowledge (“Neither the operator nor the function family is new,” in Introduction section), so its value rests on showing that conditioning the target on the old policy matters in practice. The evidence does not establish this. Proposition 1 holds by construction: a target computed from Q alone cannot depend on the old policy, but this does not show that it should. Empirically, the residual linear policy realizes an expected-Q gain of only +0.0053 against a target gain of +0.0638.
Questions for authors

None

Reviewer_459QOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Distributional Decision Revision (DiRe), which preserves the incumbent policy distribution when constructing and fitting policy improvement targets from executable feedback. Through controlled experiments, the authors argue that incumbent-relative supervision better preserves the intended policy revision than utility-only supervision or direct distribution fitting.

Critical strengths

The paper provides a systematic decomposition of policy improvement into target specification and realization. Its controlled experiments isolate the effects of incumbent information and compare multiple fitting strategies under matched conditions.

Critical weaknesses
  1. Limited Novelty. The core techniques, including KL-regularized policy improvement and residual parameterization, are well established. The main theoretical observation is straightforward and offers limited new insight.

  2. Weak Experimental Validation. The primary experiments involve only 13 development states and 59 test states with simple linear scorers. No actual LLM policy training or convincing end-to-end agent improvements are demonstrated.

  3. Poor Readability and Presentation. The paper is exceptionally difficult to follow due to excessive terminology and unnecessarily complicated explanations. The two-paragraph abstract is particularly unusual and overloaded with numerical results that lack clear context or interpretation.

Questions for authors

Could the authors clarify how the proposed approach is expected to benefit practical LLM post-training, given that the current experiments are limited to small-scale decision-level fitting?

Reviewer_sPxNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how executable feedback should be converted into supervision for policy improvement. It treats the incumbent action distribution and action utilities as a joint evaluated decision, and distinguishes constructing an improvement target from realizing that target with a finite model. DiRe instantiates this view using an established KL-regularized target and an incumbent-relative policy parameterization. Experiments on LoCoMo decision states examine target invariance, reference-dependent fitting, and the gap between intended and realized utility. Additional analyses include displacement-matched controls, richer semantic features, and an in-panel MiniWoB diagnostic.

Critical strengths

The strongest aspect is the careful separation of target construction from finite-model fitting. The fixed-target comparisons show that an update with positive intended utility can lose that benefit during realization, and the paper examines this with several relevant controls rather than relying on a single aggregate result.

I also appreciate the reporting of evidence that qualifies the proposed interpretation: the inconclusive displacement-matched comparison, heavy forward-KL tails, and prior exposure of the evaluation conversations are all stated explicitly. This makes the narrow empirical findings more credible.

Critical weaknesses

Proposition 1 states that a mapping whose only input is Q cannot distinguish incumbents sharing that Q. This is correct, but largely follows from the restriction imposed on the mapping. The corollary bounds error relative to intended revisions already assumed to differ; it does not establish a consequence for executable utility. The paper appropriately acknowledges that the improvement operator, residual parameterization, and separation of improvement from fitting are established.

The main substantive contribution therefore rests on the finite-fitting experiments. Here the strongest proximity control remains inconclusive: after matching displacement, the utility advantage is +0.00354 with a 95% interval of [−0.00045, +0.01012], and the target-JS difference also crosses zero (Table 4). Table 19 further shows that 29 of the 59 states require no revision, while residual fitting has worse target JS on the higher-revision stratum. These are useful diagnostics, but they do not yet establish a distinct benefit of the proposed interface principle beyond preserving a useful incumbent during fitting.

The primary scorer is fitted and selected on the same 13 states, and the 59 evaluation states come from three previously used conversations. The reported state-bootstrap intervals do not account for within-conversation dependence (Appendices B.4, B.5, and C.10). The 50.9% realization result uses 24 new states from another development-exposed conversation and changes the features and fitting setup without a matched direct comparator. The broader retrospective analysis realizes 13.0% of the intended gain and has worse mean target JS than retaining the incumbent unchanged (Table 18). I recognize that the authors disclose these limitations; they nevertheless leave the practical significance uncertain.

The manuscript also needs a concrete account of the primary experiment. It does not clearly identify the incumbent and continuation models, the meanings of the nine candidate slots, the original linear scorer's features, or the reward function underlying Q. Appendix B describes the general interface contract, but these implementation choices are central to understanding why direct fitting fails and what the reported utility gains mean.

Questions for authors

What are the nine candidate actions, how is the incumbent distribution obtained, which model or procedure supplies the fixed continuation, and what reward metric defines Q? Please also specify the input features of the primary linear scorer. These details are needed to interpret the realization gap.

Appendix C.12 defines the statistic using the maximum target probability divided by the maximum policy probability over the effective target support. These maxima can correspond to different actions. For example, q* = (0.999, 0.001) and π = (0.001, 0.999) give a ratio of 1 despite severe underallocation to the intended action. Is the displayed formula a typographical error or the implemented metric? This affects the interpretation of the zero severe-miss rates in Table 17 and the semantic-scorer evaluation.

Is a matched direct-versus-residual comparison available for the richer semantic scorer, using identical features, data, and model-selection procedures? The current 50.9% result demonstrates improved realization in that setting, but does not establish that retaining the incumbent as an explicit reference is responsible for it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.