- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The papers motivation is that in model based reinforcement learning standard prediction loss might not give us a good estimate of how good the downstream performance of the trained policy would be. The authors propose that using the average signal-to-noise ratio is a good measure and that policy gradient algorithm that observe high SNR (ESNR in this work) are expected to have a higher downstream performance.
The authors try out a small toy problem with discotinuity in dynamics w.r.t actions. they traine reconstruction based and reconstruction free model based RL (MBRL) methods and different policy training methods and evaluate and compare the ESNR with the downstream control performance.
The policy's downstream performance broadly agrees with ESNR with positive correlation. and the authors are honest about when ESNR can be a usefull measure and when it is not (app. D)
Critical strengths
- I do believe that with advent of world models a good and cheap estimate of how good downstream policy learning would be is very important before we throw training compute to actualy learn the policy and is also well supported by existing theory in model-erro and its downstream effects to policy learning.
- On experiments with larger vision envocers such as Dino based and Jepa based encoders used in the paper, the ordering of downstream performance and ESNR is consistent which is a strong positive.
- ESNR computation is cheap and easy to implement.
- The paper is written in a relatively straightforward manner
Critical weaknesses
-
The role of ESNR is not consistent: The authors claim a good world model for policy learning should be both accurate and have high ESNR, yet the authors also claim that ESNR is a poor metric if the model accuracy is low. these seem to be conflicting claim and I am unable to consolidate the very precise motivation. from my perspective, it seems that ESR is a good metric you should also look at but not completely rely on.
-
As an RL practitioner I would always be interested in which backbone/enocder or existing model should I use to finetune and then train my donwstream control policy, but it is unclear whether for the same policy training algorithm whether ESNR is usable as a metric across architectures?
-
The action distribution which is used to evaluate ESNR is neither the data collecting policy not policy that we are aiming to train, the "probe" uses a zero mean gaussian action distribution and it is unclear whether beyond toy settings, this distribution is good enough to give a good enoguh estimate of the ESNR
-
Limited novelty: ESNR already comes from Parmas et al and PWM (Georgiev et al) already have shoed that regularized world models yield more reliable gradients.
Questions for authors
- Algorithm 1 (line 13) weights each step’s score by the immediate reward rather than the return-to-go, and the accumulated return J is never used. Is this the estimator used in the experiments?
- Does the ranking hold when ESNR is computed on dataset actions, or on actions from the policy being optimized, rather than on zero-mean Gaussians?
- How should ESNR be compared across methods, given that each CEM sample aggregates many rollouts while each FoG or ZoG sample uses one? Does normalizing by the number of model rollouts per sample change the ranking in Experiment 1?
- For fast-wam: Why should a higher SNR of the held-out action-loss gradient at the final checkpoint indicate a better policy? A consistent nonzero mean gradient on held-out data could also reflect systematic error.
- How does the ESR score change as the amount of compute and data used to train the model increases. I am assuming the ESR increases monotonically and so does policy performance but is this empirically suported as well.
- If in one short motivation, how do the authors expect the ESR to be used, from my read I am a bit confused if it is a pre-evaluation diagnostic for policies trained using the model, or is it to give an estimate of downstream learnd policy before we start to train the policy?