- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces ASTRO, an offline model-based reinforcement learning method that uses local data geometry to guide model training and synthetic rollout generation. It constructs a support tube around each encoded next state by intersecting a Mahalanobis ellipsoid, estimated from nearby dataset points, with a relative density constraint. A robust value-aware loss moves the model's predicted next-state value toward the smallest target value among samples from this tube. The method also combines geometric distance, ensemble disagreement, and an error-calibration term into a distrust score that controls rollout length, while a SAC backbone learns from real and generated transitions. The experiments cover 24 D4RL datasets from Adroit and MuJoCo; ASTRO has the highest reported mean on 12 individual datasets. The paper additionally presents component ablations, geometric visualizations, and a theorem intended to explain an advantage over scalar trust regions.
Critical strengths
-
The motivation addresses a real difficulty in offline model-based RL: equal Euclidean distances from the dataset can represent very different degrees of extrapolation. Using nearby data to distinguish directions with substantial variation from directions with little coverage is a reasonable way to make this distinction concrete.
-
I like the decision to use the same local geometry in both the model objective and rollout generation. This gives the method a coherent organizing idea: the regions that influence pessimistic model training also influence how far the policy is allowed to roll out through that model.
-
The main comparison has useful breadth within continuous-control offline RL. It includes manipulation and locomotion, as well as expert, human, cloned, replay, and random data sources. The use of one stated hyperparameter configuration across all 24 datasets is practically appealing.
Critical weaknesses
-
Some steps in the theoretical comparison would benefit from clarification. In particular, I am not sure how the conditional-mean accuracy in Lemma F.1 translates into control of nonlinear value expectations without further distributional assumptions. Lemma F.2 appears to compare a uniform ball with the true transition law, and its connection to the robust objectives of the ball and tube methods is not entirely clear. I also could not see how the acknowledged radius-dependent pessimism bias is accounted for in the final bound. Making these connections more explicit would help clarify the scope of Theorem 3.1.
-
The quality of the support tube depends on the learned representation and its decoder. Geometrically nearby latent points need not correspond to nearby, dynamically meaningful raw states, and an apparently supported latent interpolation may decode to an implausible configuration. The conclusion acknowledges dependence on encoder quality, but this limitation also affects value evaluation and policy training through decoded model rollouts.
Questions for authors
Please see the weaknesses