acceptodds
Under review as a conference paper at ICLR 2027

Representation Sufficiency for Closed-Loop Control: Decision Risk and the Occupancy Bridge

Abstract

Learned representations for robot control are often evaluated through action prediction on demonstrated states, yet accurate prediction need not imply effective closed-loop control. We separate return loss into the information deficit imposed by a representation and the excess loss of a learned policy within the resulting policy class. On a fixed state distribution, action-value gaps bound the minimum decision risk in terms of Bayes optimal-action error. Relating this risk to attainable return requires accounting for the different state occupancies induced by the risk-minimizing decision rule and the best policy available under the representation. A localized correction further tightens the bounds. Under a fixed macro-action interface, removing direction information reduces the exact optimal discounted return from to ( per macro decision). The resulting information deficit is , whereas decision risk under expert occupancy, scaled to the same return units, is only . Since both optimal returns are computed under the same interface, the deficit reflects the representation restriction rather than learned-policy excess. We further show that identical recorded demonstrations can be consistent with learned-policy excess ranging from zero to linear in the horizon, even when the representation permits optimal control. Finally, fixed-interface data aggregation improves state-based, frozen-latent, and visual policies without adding information at execution time, providing evidence that part of the learned-policy excess is recoverable within the same policy class.

Primary area:
reinforcement learning
Keywords:
Representation Sufficiency, State Abstraction, Closed-Loop Control, Distribution Shift, Imitation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_NfwROpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

I am less familiar with the theory behind this work, so my review focuses mainly on the motivation and experimental evidence.

The paper studies why good action prediction does not necessarily lead to good control. It distinguishes two sources of performance loss: information missing from the representation, and a learned policy failing to use the information available to it. The bounds relate decision errors to achievable return while accounting for changes in the states visited by the policy. A separate result shows that demonstrations alone may not reveal how far a learned policy is from the best policy using the same representation. The experiments use small finite-state models, exact calculations on restricted robot simulations, and learned-policy studies with data aggregation.

Critical strengths
  • It is useful to ask separately whether a representation contains enough information and whether a learned policy uses it well. This helps avoid blaming the representation for a failure to learn a good policy (Section 3).

  • The exact graph calculations compare the best achievable returns for different representations, so the gap is not simply due to poor training. Reporting the loose bounds and sensitivity to policy choices also helps interpret the results

Critical weaknesses
  • The robotics evidence is too limited to establish practical value. The exact study uses four development base blocks and 20 fixed macro actions, and the learned-policy experiments are also in simulation. Without real-robot validation, it is unclear whether the analysis helps explain failures under physical sensing, contact, and execution variability.

  • The tighter bounds still do not give a positive lower bound on representation loss in the robot example. Every coarsened representation in Table 2 has a negative lower endpoint, giving zero after clipping. This remains true even after choosing among return-optimal policies to minimize the localized correction (Appendix B.5). Since the calculation also needs oracle values and policy occupancies, it is unclear what practical benefit the refinement offers in this setting.

Questions for authors
  • What existing evidence supports the framework’s diagnostic value for physical robot failures or beyond the four development blocks?Which training/evaluation results are available to verify that evidence?
  • Under what conditions does the localized refinement provide a nontrivial lower bound, and does any evaluated robot setting satisfy them?
Reviewer_FsQtOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I have read this paper over several times and still have yet to understand what it is contributing. The paper is full of seemingly AI-generated results with incredibly poor positioning, no relation to prior work, and verbose, inexplicable statements with LLM pseudo-babble. The authors effectively define a "representation" as a form of partially-observed MDP but do not make any acknowledgement or connection to partially observed MDPs. The experiments are similarly incomprehensible.

Critical strengths

Representations are indeed critical for learning policies. The authors have effectively defined a POMDP and declared the observation a "representation." There is value in understanding what observations are needed for learning control policies and extensive existing literature on hardness results and gaps arising from MDPs vs POMDPs but I don't believe this paper makes any comprehensible contributions in this regard.

Critical weaknesses
  • The introduction gives no indication of what is being studied beyond "representations." The paper is completely unintelligble, starting with Figure 1 (which seems to have a Comic Sans title and random unexplained experimental numbers in it for some reason?). Even the abstract itself contains statements like "Under a fixed macro-action interface, removing direction information reduces the exact optimal discounted return from 0.990 to 0.291" which makes no sense as a context-less statement in the abstract and makes no sense as a statement to give in the abstract.
  • R is essentially just an observation map. The authors are effectively just studying an POMDP, but frame it as an MDP with a "representation map." This is not properly situated with respect to the prior literature.
  • Terms are defined and never used. Critical terms (such as what is "discounted policy occupancy") are never defined.
  • Lemma 1 having a "proof" that is deferred to the appendix is bizarre given that the entire statement is of the form "a - b = a - c + c - b" and is entirely a consequence of adding/subtracting a J(π⋆)J(\pi^\star) term. Lemma 1 is just a hardness gap between partial observation and full observation in MDPs, which is well known and not novel. There is no positioning with respect to any prior work in POMDPs.
  • The experiments are similarly incomprehensible. The jargon used to describe the environment "direction cells," "base identity," and "public-action prefix" are not explained and seem to be hallucinated. What is H2 and H3? Are these hypotheses? Is there an H1?
  • What are LGL_G, LWL_W, LDL_D shown in the table? The caption text is incomprehensible and for the primary measured endpoints having a half-sentence jargon-filled explanation of what they are is not acceptable.

I could list another dozen issues with this paper, but I think the above alone justify outright rejection without further reconsideration.

Questions for authors
  • Please position your work appropriately with respect to POMDPs and prior literature.
  • What is going on in Figure 1?t
  • Please explain what LGL_G, LWL_W, LDL_D are.
  • What is your experimental setup? What does "The robot finite processes are induced from robomimic Lift snapshots" mean? What are "development base blocks"? Lift is a single standard task in a robomimic environment. Why is the robot initialization held fixed? Where does the training data come from if not from differing initializations?
Reviewer_FRZiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the relationship between representation-level decision quality on a fixed distribution and attainable closed-loop return. The authors decompose return loss into an information deficit imposed by the representation and the excess loss of a learned policy relative to the best representation-constrained policy. The results show that, under the specified interface, low decision risk under expert occupancy can coexist with a large information deficit.

Critical strengths
  1. The paper clearly distinguishes representation limitations from policy-learning failures. Section 3 separates the best control performance permitted by a representation from the performance achieved by a particular learner, avoiding the direct attribution of learning failures to insufficient information. This distinction provides a useful framework for understanding the gap between action prediction and closed-loop control.

  2. The exact comparisons and sensitivity analyses within the finite interface are thorough. Section 4.3 compares full-state and representation-constrained optimal policies under the same macro-action interface, removing differences in training as a confounding factor. Appendix B further examines policy completion, terminal-state encoding, and numerical consistency, strengthening the internal validity of the risk–return separation demonstrated in these finite instances.

Critical weaknesses

All localized lower bounds for coarsened representations in Table 2 are negative. Even after further optimizing policy selection, the blind-representation lower bound remains −0.4503 (Table 6). In Table 5, the localized upper bound for the blind representation is 1.0794, while an alternative risk-minimizing rule yields 1.0521; both exceed the natural upper bound on the information deficit in this task. Therefore, after clipping to the natural range, the reported localized interval for the main blind-representation instance does not narrow the a priori admissible range.

Questions for authors
  1. Relative to the cost-sensitive decision analysis, performance-difference identity, and discrepancy bounds already cited, what is the paper’s core new theoretical result? Which conclusions about representation sufficiency cannot be obtained directly from existing tools?

  2. All coarsened lower bounds in Table 2 are negative, and the localized blind-representation upper bounds in Table 5 exceed the natural return range. Given the reported results, which otherwise admissible information-deficit values do these bounds actually rule out? What do these results establish as the framework’s primary use?

  3. Without first solving for the representation-constrained optimal policy, can the methods currently presented in the paper still provide meaningful benefits? Please distinguish guarantees already established in the paper from issues left for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.