acceptodds
Under review as a conference paper at ICLR 2027

Decision-weighted prediction redirects what a capacity-limited encoder retains

Abstract

Capacity-limited predictive encoders are standard in model-based control: each compresses a controller's belief into a code too small to hold it, and no guarantee says the code keeps the directions the control cost depends on. The unweighted objective ranks belief directions by predictive energy, which the control cost does not enter. We prove, using standard spectral arguments, that one reweighting of the target redirects the same capacity onto the directions the cost weights most. The weight is a fixed matrix, the factor that makes prediction error measure control cost. That statement is read at an exact optimum with no tie in the cost's ranking, so our finitely trained runs illustrate it rather than test it. Such a factor exists exactly when the predictive map discards no decision-relevant direction; a counterexample shows the reweighting cannot be dropped. Two companion results drop encoder linearity, show decoder linearity cannot be dropped, and make the condition checkable in advance; every theorem is machine-checked in Lean. On synthetic linear-Gaussian systems the weighted target lowers decision-relevant error wherever the capacity constraint binds.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, world models, model-based reinforcement learning, partial observability, JEPA, optimal control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QtJ7OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper seems to be all written by AI. It is not human readable. After much effort, I am not able to understand what is the main problem of interest and what are new contributions. The writing mixes lot of topics, without a clear message even for the abstract and introduction parts. The paper was done by using about 11 modern AI tools/agents.

Critical strengths

A comprehensive collection with 78 pages containing both maths and empirical results.

Critical weaknesses

This paper seems to be all written by AI. It is not human readable. No new contribution seems to be introduced.

Questions for authors

NA

3 comments
AuthorsOpenReview ↗

Response (1/2): the paper in brief

Thank you very much for your review and for the effort you put into reading our paper. We are very sorry that you found it hard to read, and we would like to make it clearer.

The paper in brief. A model that learns to predict what an agent will observe often has to compress what it knows into a few numbers. In the linear setting we study, the best such model keeps the directions with the most predictable variation. These need not be the directions the control cost depends on. We prove that multiplying the prediction target by one fixed matrix fixes this: at the exact optimum, and under the conditions in our second comment, the model then keeps only directions the control cost depends on. The matrix is computed from the system, the controller and its cost, and we state exactly when such a matrix exists. Most experiments illustrate this on linear systems. A second family (§4) checks what the weight presumes: that a model predicting from a window of past observations recovers the controller's state estimate at all.

The idea, in Figure 1. Figure 1 shows one pipeline trained two ways: an encoder compresses past observations into a code of k numbers, and a decoder predicts the next observations from that code. Trained the standard way (a), the code keeps the directions with the most predictable variation, whether or not the controller needs them; trained on the prediction target multiplied by our matrix (b), its optimum, under those conditions, keeps only what the controller needs. Our pixel experiment shows this on camera images of the DeepMind Control Suite's cartpole next to a second pendulum the task does not depend on, whose noise we set so that, in the linearisation, the standard optimum lies on that pendulum (Appendix I.14, p. 106). The standard code is mostly about the second pendulum (a median 0.83 of the part of its variance that the two pendulums' states explain); the code trained on a truncated form of the matrix (training on the exact one failed there) almost not at all (0.004), and its error on what the controller needs is lower on 20 of 20 seeds.

Where were the difficulties? If you could tell us which passages, terms or steps were hard to follow, we will rewrite them. If you have a concern about the soundness of a particular result, we would be grateful to hear which one, so that we can address it.

AuthorsOpenReview ↗

Response (2/2): what we prove, what is new, and how the paper was written

What we prove, more precisely. Let the matrix T map the controller's state estimate to the expected future observations, and the matrix D give how the cost depends on the state.

  1. A weight V with VT = D exists exactly when every direction that leaves the expected observations unchanged is also ignored by the cost (in symbols, ker T ⊆ ker D); V = DT⁺ is one such weight (§3).
  2. With it, at the exact optimum and under Theorem 2's conditions on k and on a gap between the last kept and the first discarded eigenvalue, the best code of k numbers keeps only directions the cost depends on.
  3. Without the weight, the best code can keep a direction the cost ignores (Figure 1(a) sketches such a case).
  4. With a linear decoder, allowing nonlinear encoders does not change the optimum, and at that optimum the decoded output depends on the state estimate only through D (Theorem 3, which assumes the estimate takes finitely many values). A nonlinear decoder, at the same capacity, can reach a lower loss than any linear one while depending on a direction the cost ignores (Appendix A.3).

What is new. The tools are classical, as the paper says: Ky Fan's maximum principle and the Eckart-Young-Mirsky theorem for which directions a code keeps, and Douglas's range-inclusion theorem for when the weight exists. What we add is their use for capacity-limited predictive models: the existence condition joined to the statement of what the code keeps, a counterexample, the extension to nonlinear encoders, and a check of the condition before training.

How the paper was written. This paper is the work of several authors, and AI assisted different parts of our process to different degrees. Everything produced with AI assistance went into the paper after careful and meticulous human oversight: the authors read, checked and revised every part, several times and in detail. We also built the process so that the results do not rest on the text: every theorem is machine-checked in Lean and the paper reproduces its statements from the Lean sources, every reported number was checked against the recorded experiment results (the revision corrects the printing errors later checks found), and the Lean development and the code behind every number are in the supplementary material, so the results can be confirmed independently of the written text.

If the review's "about 11 modern AI tools/agents" refers to our AI Use Statement (pp. 10 to 11 of the submitted PDF, 11 to 12 of the revised one), that list names every model we used at any point, counting successive versions of one model separately (Claude Opus 4.6, 4.8, 5 and 5.5 are four of the eleven), to be complete and precise. It is not a measure of how much of the work AI did. Claude models, above all Claude Opus in its successive versions, did most of the AI-assisted work; the models of the other two providers had smaller, specific roles, which the statement names.

I thank the authors for providing further explanations and revised the paper heavily.

My concerns remain. I've just seen the revised paper. It seems that the authors did extensively revise the PDF, from 78 pages to 113 pages. However, it is unclear about what have been changed and why they need significant changes?

The writing is very weird. For example, Lines 173--239 contain only one paragraph to present both Theorem 2 and possibly discussions. I even cannot see the signal that terminates Theorem 2 statement.

Therefore, I keep the current rating.

Reviewer_vD1oOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a decision-weighted prediction objective for capacity-limited predictive encoders. The authors characterize when the weighting can align the information retained by an encoder with downstream decision relevance, provide a counterexample for unweighted prediction, and formally verify their theoretical results in Lean. Experiments on synthetic systems illustrate the proposed approach.

Critical strengths
  1. Clear theoretical motivation. The paper makes a good case for why prediction accuracy alone may not be enough when the goal is downstream decision making. The counterexample illustrates how an encoder can spend its limited capacity on information that does not matter for the decision.

  2. Careful treatment of assumptions. The paper is explicit about the conditions under which its results hold and discusses limitations such as the linear decoder assumption. The formal verification in Lean is also a useful addition.

Critical weaknesses
  1. The novelty relative to prior work is not entirely clear. The paper builds on existing ideas in decision aware representation learning and low rank approximation. While the authors bring these ideas together in the setting of capacity limited predictive encoders, I was left wanting a clearer explanation of what is new compared with the closest prior work.

  2. The practical benefits remain unclear. The proposed weighting requires information about the system dynamics and decision costs, which may not be readily available in practice. The experiments are also primarily on synthetic linear Gaussian systems, with one additional nonlinear plant experiment. I would have liked to see more evidence that the approach works when these quantities are estimated and the dynamics are more realistic.

Questions for authors
  1. How does the proposed characterization differ from existing work on value directed compression and goal oriented low rank approximation? It would be helpful to clarify what new insight the paper provides beyond applying these ideas to capacity limited predictive encoders.

  2. In practice, the system dynamics and decision costs may not be known exactly. How sensitive is the approach to errors in the estimated weighting, and how could the weighting be obtained when these quantities are unknown?

3 comments
AuthorsOpenReview ↗

Response (1/2): Q1 and W1, what is new; Q2, sensitivity to errors in the weight

Thank you very much for your review and for the two insightful questions.

Question 1 and weakness 1. What is new against value-directed compression and goal-oriented low-rank approximation. Both compute the best compression for a goal they are given. Our question is a different one. Many world models are already trained to predict their observations, and the goal plays no part in that objective. We ask whether a fixed weight on that objective can make its error measure the goal instead, and we prove exactly when it can. We then prove that under such a weight the capacity-limited optimum keeps only the directions the goal depends on. By Theorem 2, such a weight exists exactly when ker T ⊆ ker D; at a capacity within the decision rank and under a spectral gap, its optimum keeps only decision-relevant directions; without it, the optimum can spend its capacity on directions the cost ignores (decision-irrelevant energy falls from 0.707 to 0.00463 at k = 2; §5). Theorems 3 to 5 extend this to nonlinear encoders, a check before training, and near-optimal codes, all machine-checked in Lean. Given the weight, the kept subspace is classical, and the paper says so.

The revision compares both of your references in §2 (pp. 2 to 3): value-directed compression's fixed-policy linear analogue is (ker D)⊥ once the cost horizon reaches d − 1 (A.2, pp. 26 to 27; Proposition 9), and goal-oriented low-rank approximation acts on the parameters directly; Proposition 10 (I.15) states, under its eigenvalue hypothesis and up to the decision rank, exactly when a subspace ranked as LQG balancing ranks it agrees with ours, and what keeping it costs; and, computed from each system's exact matrices, at k = 2 our subspace lies more than 10° from eleven other classical recipes on 92 to 100% of the drawn d = 8 systems on which each recipe is defined (I.3).

Question 2. How sensitive is the approach to errors in the weight? In a controlled perturbation (one cell), little: at relative errors of 0.024 to 0.23 in the matrix the weight is built from, decision-relevant error rises by 0.006 to 0.020 on a base of 0.669, while the code's share on decision-irrelevant directions rises from 0.040 to 0.244 (H.14, pp. 73 to 74). Theorem 2 clause 4 bounds what an inexact weight retains under a spectral gap, and the revision adds a machine-checked bound with no gap: if the weighted Gram operator is within ε of the exact one in operator norm, a capacity-k optimum keeps all but at most 2kε of the top-k decision energy (Proposition 11, I.16, p. 108).

AuthorsOpenReview ↗

Response (2/2): Q2, obtaining the weight when dynamics and costs are unknown, W2, more realistic settings

Question 2. Obtaining the weight when the dynamics and costs are unknown (I.1, pp. 81 to 82). From observations, actions and per-step costs, with no state and no system matrix (horizons, gain and the cost's quadratic form granted): 0.966 of the exact weight's benefit from 4 trajectories, 0.997 from 64 (d = 16). From a running controller's logs alone (no state, no system matrix, no gain; horizons granted): better than no weight on 20 of 20 seeds; an estimator that picks its rank from data can do worse than no weight.

Weakness 2. More realistic settings:

  • A control-suite cartpole, balanced in closed loop: with eight distractors in the observations, the LQR gain on a four-number linear code's readout keeps the pole up on 20 of 20 seeds with the weight, exact or fitted from each run's own data; without it the pole leaves upright in every episode. The gain, the cost, the readout's fit and the controller that generated the training data come from the simulator (I.19, pp. 112 to 113).
  • The nonlinear pendulum in closed loop: the weighted code keeps 0.86 to 1.02 of the controller's saving at three noise levels, against 0.30 to 0.53; with the weight fitted from recorded states (B, gain, cost and horizons granted), 0.69 to 1.02; identified without the state (gain, horizons and the cost's form granted), more than no weight in three of six registered comparisons, none significantly less (I.2, p. 83).
  • Pixels: on dm_control's cartpole with a distracting second pendulum, the code trained with the truncated weight holds a median 0.004 of its state-explained variance on the distractor against 0.83, with lower decision-relevant error on 20 of 20 seeds, and a weight built from dynamics fitted to each seed's own trajectories gives a similar improvement; the exact weight did not train there, and no controller on any of these codes kept the pole up (I.14).
  • 64-dimensional systems: lower error on 20 of 20 seeds at k = 2, 4 and 8 (I.17).

Where it does not carry over: on cartpole swing-up with a learned world model and a planner, the weighted models keep more of the plant and almost none of the distractors, but do not detectably raise the return, and the exact weight's return is lower in seven further cells; that weight, built at upright, is zero on the pole cosine, which tells upright from hanging (I.17, pp. 110 to 111). On a linearised three-pole cartpole, training on the exact weight ends worse than no weight (§7).

Thank you for the detailed revision. The clarifications and added experiments are helpful, though the broader applicability of the approach remains unclear. I will retain my original rating.

Reviewer_sQr9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how capacity-limited predictive encoders can prioritize highly predictable directions while discarding information important for decisions. It constructs V=DT+V=DT^{+}, with exact realization VT=DVT=D possible iff ker⁡T⊆ker⁡D\ker T\subseteq\ker D. Detailed synthetic experiments show lower decision-weighted error, with benefits largely disappearing at full capacity.

Critical strengths
  1. The realizability condition distinguishes information missing from the prediction target from information lost through compression, providing a useful diagnostic before training.
  2. Detailed matched-capacity comparisons, loss-scale normalization, and learning-rate sweeps support subspace reallocation as the mechanism, beyond simply changing model size or optimization scale.
Critical weaknesses
  1. Much of the theory combines classical factorization and weighted reduced-rank regression. A distinct statistical or algorithmic advantage over decision-target learning under matched information access remains unestablished.
  2. The fitted-weight experiment still requires measured states and known cost/controller parameters; naïve pseudoinversion amplifies noise and performs worse than unweighted training.
  3. The guarantees critically depend on a linear decoder, substantially limiting their applicability to nonlinear latent world models. Allowing nonlinear encoders leaves this central restriction intact.
Questions for authors
  1. Since VT=DVT=D can admit multiple factors, why choose DT+DT^{+} rather than minimize target-noise variance under the same constraint?
  2. Can you derive DD from the excess control cost caused by deploying the compressed belief, and demonstrate closed-loop improvements? Appendix A.2 indicates that the current metric measures a different quantity.
  3. Given the strong assumptions of known predictive and decision operators, linear decoding, and exact population optimality, can you demonstrate a concrete advantage over direct decision-target learning under matched data and model information?
4 comments
AuthorsOpenReview ↗

Response (1/3): Q1, why DT⁺; W2, measured states and known parameters

Thank you very much for your review and for three precise questions. We answer each with new results in the revised PDF.

Question 1. Why DT⁺ rather than the least-noise factor? For the optimum the choice does not matter: every exact factor changes the objective only by a constant, so all share its exact optima (linear encoders). The least-noise factor is the generalised least-squares (GLS) factor for the covariance of the target's residual given the window, and DT⁺ is the least-noise factor for every realizable D exactly when the noise covariance leaves range(T) invariant, as isotropic noise does (Proposition 2, I.9, pp. 86 to 87, machine-checked in Lean). You are right that VT = D fixes V only on range(T); our appendix sentence saying it fixes V up to a left orthogonal factor was wrong (corrected in H.12, p. 68). Estimated on the d = 16 cell from 8 trajectories with the state recorded, the GLS factor lowers decision-relevant error about as much as the true DT⁺; on the cartpole of I.7 this factor is worse than no weight, estimated or computed from the true system (I.9).

Weakness 2. Measured states and known parameters. The revision builds the weight without measured states (I.1, I.2, pp. 81 to 83). From observations, actions and per-step costs, with no state and no system matrix (horizons, gain and the cost's quadratic form granted), it keeps 0.966 of the exact weight's benefit from 4 trajectories and 0.997 from 64 (d = 16, k = 2). From a running controller's logs alone (no state, no system matrix, no gain; horizons granted), it beats no weight on 20 of 20 seeds; an estimator that picks its rank from data can do worse than no weight. On the nonlinear pendulum in closed loop, identified without the state (gain, horizons and the cost's form granted), it keeps more of the controller's saving than no weight in three of six registered comparisons, none significantly less. Truncating the estimated pseudo-inverse, which amplifies noise as you say, at its largest relative spectral gap removes this on the d = 16 cell (I.7); on the cartpole cells chosen in advance, the regularised estimate trained significantly worse than no weight in three of four cells and the truncated one in all four (I.11), and I.17 reports the gain beside the weighted target's noise-to-signal ratio, computable before training from the system matrices and the noise covariances, in two cells that do not isolate it.

AuthorsOpenReview ↗

Response (2/3): Q2, a weight from the excess control cost and closed loop; W3, the linear decoder

Question 2. A weight from the excess control cost; closed loop. Yes. Under a fixed gain an error changes the cost through the input it changes: for one isolated error this gives the operator M^{1/2}K_c, plus a cross term with the state that vanishes in expectation for an error uncorrelated with the state and for every error under an LQR gain. New Proposition 12 (I.18, machine-checked) gives the exact finite-horizon excess, including the terminal value x_NᵀP_cl x_N, of an error at every step: under an LQR gain, Σ‖M^{1/2}K_c e_t‖² plus a noise term whose zero mean for causal estimates is argued in prose; under the paper's gain two cross terms remain, so for our codes it is approximate. The theorems apply to it as to any linear D, and it is realizable wherever D is (I.10, p. 90). As you note, the submitted metric (A.2) is a different quadratic form.

In closed loop (I.2, I.10, p. 90), codes trained on the action-map weight K_cT⁺ keep more of the LQG saving under K_c than unweighted codes in all four partially observed cells, and under K_c also beat codes trained on our cost-to-go weight; codes trained on our weight at k = 4 cost less than unweighted codes under three controllers (the paper's fixed gain, a gain optimised for each code, and an LQR designed on each code's fitted model with its last 8 actions); on the nonlinear pendulum the weighted code keeps 0.86 to 1.02 of the controller's saving at three noise levels against 0.30 to 0.53 (in cost, significant by Wilcoxon, not by t-test). On a control-suite cartpole with distractors, the simulator's LQR gain on a linear code keeps the pole up with the weight, exact or fitted, and not without it (I.19). I.2 and I.10 also report where this does not hold: under full observation, at k = 2 under the paper's gain on open-loop data, without the action history (every designed loop fails and ours costs more), against value equivalence on the Kalman belief under an optimised gain, with exploratory actions under an LQR gain, and on the pendulum's linearisation.

Weakness 3. The linear decoder. We agree that the guarantees need it: Theorem 3 drops encoder linearity and shows that decoder linearity cannot be dropped in general (clause 5, A.3). Empirically, with an MLP decoder the weighted code's decision-relevant error is 0.814 against 0.998 unweighted (k = 2; one cell, where the optimal decoder is already linear; D.6). A pixel encoder on dm_control's cartpole with a distractor pendulum, whose noise we set so the unweighted optimum lies on it, puts a median 0.83 of its state-explained variance on the distractor unweighted, 0.004 with the truncated weight, with lower decision-relevant error on 20 of 20 seeds; the exact weight did not train there, and no controller on any of these codes kept the pole up (I.14). A learned world model on swing-up keeps more of the plant with the weight but does not detectably raise the planner's return, and the exact weight's return is lower in seven further cells (I.17).

AuthorsOpenReview ↗

Response (3/3): Q3 and W1, advantage over learning the decision target, the classical parts

Question 3 and weakness 1. An advantage over learning the decision target directly. We did not find "decision-target learning" defined, so we tested each reading we could, every objective at equal capacity, and most in I.12 at their own tuned step sizes:

  • Learning a decision target from the same data, without the Kalman belief: for the objectives we tested, mostly yes. Our weight trains lower than a TD-MPC2-style objective and DBC, both given the true stage cost, in all 7 settings, and lowest of nine tuned objectives on the nonlinear pendulum (ours built from its linearisation, the others from simulator outputs alone), with a cheaper loop in 31 of 32 comparisons. VaGraM's diagonal is slightly above ours with linear encoders and below with a recurrent one; at k = 2 the TD-MPC2-style and DBC codes give cheaper loops in all three settings (§6); regressing logged noise-free cost outputs and actions, which shares our optimum, trains slightly lower where the cost reads observed channels (logs of a controller on the exact Kalman belief; I.1).
  • Value-aware model learning: at k = 2, on data with recorded actions, ours trains lower than robust VAML and IterVAML, though their loops cost less under an LQR gain except with a generic value basis; VAML with our weighted target's values shares our optimum and trains slightly lower.
  • Value equivalence on the Kalman belief: that belief needs the noise covariances, and our weight does not. With a misjudged or misspecified noise model it can fall behind ours. With the true covariances it shares our optimum, and its lead is in training; with fitted covariances it still trains lower than ours (§5, I.12). Under matched information on the nonlinear pendulum (the same records and identified model for every arm, no true state or system to learn from), the closed loop on our weight's code costs a median 13.7% to 15.1% less than on either fitted direct target's, on 18 to 20 of 20 seeds, as registered; on the linear plant its counterpart keeps the lower decision-relevant error (I.12, p. 100).

Propositions 7 to 9 (pp. 100 to 106, machine-checked) explain part of this: under their stated assumptions, a causal linear filter fixed in advance, linear reward and value heads, and codes that respect the coarsest bisimulation can each fall short of our exact optimum. We have not compared with decision-targeted digital twins (Amad and van der Schaar, ICML 2026), cited in §2 as concurrent work.

Weakness 1. The classical parts. We agree, and §1 names them. Each answers its own question: weighted reduced-rank regression says which directions a rank-k optimum keeps under a weight that is given, and the factorization says when VT = D has a solution. Neither says which weight to put on the prediction target, the observations a world model already learns to predict, so that a capacity-limited optimum keeps the directions the control cost depends on rather than the most predictable ones. Theorem 2 answers that: a weight that makes prediction error measure decision-relevant error exists exactly when ker T ⊆ ker D, that is, when the predictive map discards no decision-relevant direction; within the decision rank and under a spectral gap, the optimum under that weight keeps only decision-relevant directions; and a witness shows that the unweighted optimum can keep a direction the cost ignores.

Thank you for the detailed response and the additional experiments. However, I am still unclear about the potential applications and will maintain my rating of 2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision 1: summary of changes

We thank all three reviewers very much for their time and reviews. The revised PDF uses the 10 body pages the discussion allows. The replies to each review give details and page numbers.

  • Contributions and related work: §1 and §2 state what we add before the classical results it builds on, and §2 discusses value-directed compression, goal-oriented low-rank approximation, TD-MPC2, IterVAML and concurrent work on decision-targeted digital twins (QtJ7, vD1o).
  • New results (§5 to §7, in full in the new Appendix I, pp. 81 to 113): weights built from logged data; closed-loop cost and a decision operator from the excess control cost; trained comparisons with TD-MPC2-style learning, DBC, VAML, IterVAML and value equivalence, one of them under matched information on a nonlinear pendulum; the sensitivity to a wrong weight; a pixel encoder on a control-suite cartpole, a learned world model on swing-up, and a control-suite cartpole kept balanced by the simulator's LQR gain on a linear code; and where the weight does not help (sQr9, vD1o).
  • Proofs: eleven new propositions, and three steps the submitted appendix argued in prose (A.1, A.2, H.7), machine-checked in Lean; the prose steps that remain are named (I.10, I.18).
  • Corrections: the appendix sentence on the factor V (H.12; sQr9), a misprinted 0.00460 (now 0.00463), and smaller errors in the text (among them A.3 and C.1), tables, p-values and rounding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.