- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
I am less familiar with the theory behind this work, so my review focuses mainly on the motivation and experimental evidence.
The paper studies why good action prediction does not necessarily lead to good control. It distinguishes two sources of performance loss: information missing from the representation, and a learned policy failing to use the information available to it. The bounds relate decision errors to achievable return while accounting for changes in the states visited by the policy. A separate result shows that demonstrations alone may not reveal how far a learned policy is from the best policy using the same representation. The experiments use small finite-state models, exact calculations on restricted robot simulations, and learned-policy studies with data aggregation.
Critical strengths
-
It is useful to ask separately whether a representation contains enough information and whether a learned policy uses it well. This helps avoid blaming the representation for a failure to learn a good policy (Section 3).
-
The exact graph calculations compare the best achievable returns for different representations, so the gap is not simply due to poor training. Reporting the loose bounds and sensitivity to policy choices also helps interpret the results
Critical weaknesses
-
The robotics evidence is too limited to establish practical value. The exact study uses four development base blocks and 20 fixed macro actions, and the learned-policy experiments are also in simulation. Without real-robot validation, it is unclear whether the analysis helps explain failures under physical sensing, contact, and execution variability.
-
The tighter bounds still do not give a positive lower bound on representation loss in the robot example. Every coarsened representation in Table 2 has a negative lower endpoint, giving zero after clipping. This remains true even after choosing among return-optimal policies to minimize the localized correction (Appendix B.5). Since the calculation also needs oracle values and policy occupancies, it is unclear what practical benefit the refinement offers in this setting.
Questions for authors
- What existing evidence supports the framework’s diagnostic value for physical robot failures or beyond the four development blocks?Which training/evaluation results are available to verify that evidence?
- Under what conditions does the localized refinement provide a nontrivial lower bound, and does any evaluated robot setting satisfy them?