acceptodds
Under review as a conference paper at ICLR 2027

Anisotropic Support-Transport Robust Offline Model-based Reinforcement Learning

10 views

Abstract

Offline model-based reinforcement learning (MBRL) suffers from a structural failure mode: the learned dynamics model is accurate on the data support but drifts rapidly off-support, where the policy naturally explores. Existing remedies—uncertainty penalties, pessimistic surrogate MDPs, and adversarial model learning—reduce trust to a single scalar quantity and commit to a fixed rollout horizon, ignoring the anisotropic geometry of real offline datasets. We introduce **ASTRO** (Anisotropic Support-Transport Robust Offline RL), which replaces the scalar trust radius with a data-adaptive *support tube* carved by local *k*-nearest-neighbor covariance and density estimates. The same geometric primitive gates both a robust value-aware model-learning loss—driving predicted latent values toward the minimum over the tube—and a state-dependent rollout horizon selected by a learned trust score. On 24 D4RL datasets spanning the Adroit and MuJoCo suites, ASTRO consistently matches or surpasses strong model-based baselines, with the largest gains on support-fragmented datasets where scalar trust abstractions are least appropriate.

Primary area:
reinforcement learning
Keywords:
data-adaptive support tube, offline RL, model-based reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_X746OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ASTRO, an offline model-based reinforcement learning method that uses local data geometry to guide model training and synthetic rollout generation. It constructs a support tube around each encoded next state by intersecting a Mahalanobis ellipsoid, estimated from nearby dataset points, with a relative density constraint. A robust value-aware loss moves the model's predicted next-state value toward the smallest target value among samples from this tube. The method also combines geometric distance, ensemble disagreement, and an error-calibration term into a distrust score that controls rollout length, while a SAC backbone learns from real and generated transitions. The experiments cover 24 D4RL datasets from Adroit and MuJoCo; ASTRO has the highest reported mean on 12 individual datasets. The paper additionally presents component ablations, geometric visualizations, and a theorem intended to explain an advantage over scalar trust regions.

Critical strengths
  • The motivation addresses a real difficulty in offline model-based RL: equal Euclidean distances from the dataset can represent very different degrees of extrapolation. Using nearby data to distinguish directions with substantial variation from directions with little coverage is a reasonable way to make this distinction concrete.

  • I like the decision to use the same local geometry in both the model objective and rollout generation. This gives the method a coherent organizing idea: the regions that influence pessimistic model training also influence how far the policy is allowed to roll out through that model.

  • The main comparison has useful breadth within continuous-control offline RL. It includes manipulation and locomotion, as well as expert, human, cloned, replay, and random data sources. The use of one stated hyperparameter configuration across all 24 datasets is practically appealing.

Critical weaknesses
  • Some steps in the theoretical comparison would benefit from clarification. In particular, I am not sure how the conditional-mean accuracy in Lemma F.1 translates into control of nonlinear value expectations without further distributional assumptions. Lemma F.2 appears to compare a uniform ball with the true transition law, and its connection to the robust objectives of the ball and tube methods is not entirely clear. I also could not see how the acknowledged radius-dependent pessimism bias is accounted for in the final bound. Making these connections more explicit would help clarify the scope of Theorem 3.1.

  • The quality of the support tube depends on the learned representation and its decoder. Geometrically nearby latent points need not correspond to nearby, dynamically meaningful raw states, and an apparently supported latent interpolation may decode to an implausible configuration. The conclusion acknowledges dependence on encoder quality, but this limitation also affects value evaluation and policy training through decoded model rollouts.

Questions for authors

Please see the weaknesses

Reviewer_ncVQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ASTRO is an offline model-based RL method that uses local data geometry to guide conservative model learning and rollout generation. It constructs an anisotropic support tube using nearest-neighbor covariance and density estimates, uses this tube to define a robust value-aware loss, and selects rollout lengths through a trust score. The method is evaluated on 24 D4RL datasets covering manipulation and locomotion.

Critical strengths

1.Clear motivation and a coherent design. Using local data geometry to distinguish well-covered directions from poorly covered ones is an interesting idea. Applying this information to both model training and rollout control gives the method a clear, unified motivation.

2.Promising benchmark results. The evaluation covers 24 datasets with five random seeds, and ASTRO achieves the highest mean return on 12 datasets. These results suggest that the overall approach has practical value.

Critical weaknesses

The ablation study does not sufficiently establish the contribution of the paper’s central innovations.

The authors only conduct ablations on four datasets. Two of them, Door-C and Hammer-C, have extremely low returns even with the full ASTRO method, suggesting that the policies have not learned effective task behavior. Therefore, large relative reward differences on these tasks provide limited evidence for the importance of the proposed components. On the other two datasets, Hopper-MR and Walker-MR, where the full method achieves much higher returns, removing the RVL loss causes the largest performance drop. This raises an important concern: the performance gains may primarily come from pessimistic value learning, which is not a new idea, rather than the proposed anisotropic support tube or adaptive rollout mechanism. The current experiments do not sufficiently isolate the contributions of these components. More comprehensive ablations on well-performing tasks are needed to establish whether the paper's main innovations provide substantial improvements beyond existing conservative model-learning approaches.

I would be willing to raise my score if the authors can convincingly address my concern that the performance gains may primarily come from the RVL loss rather than the proposed anisotropic support tube.

Questions for authors

1.Can the authors provide more comprehensive ablations on datasets where ASTRO achieves strong performance?

2.Can the authors compare the proposed anisotropic support tube with a spherical support region whose radius adapts to the local data? This would help clarify whether the improvement comes from directional information or simply adjusting the size of the support region.

  1. What replaces the trust score in w/o Trust Score, and how are rollout lengths selected? How does this differ from w/o Adaptive Horizon? If both use the same fixed horizon, what other settings distinguish them?

4.How is the latent decoder trained? Please provide more detail about the decoder.

Reviewer_wD9EOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Offline MBRL fails off-support. Prior work reduces trust to a global scalar (penalty, adversarial weight, or Wasserstein radius ξ\xi) and uses a fixed horizon. ASTRO builds a local, anisotropic support tube from KK-NN covariance and density in a latent space, and reuses it for a value-aware robust loss, a trust score, and a state-dependent horizon. A bound claims the tube avoids an "anisotropy gap" of scalar balls. On 24 D4RL tasks it is best on 12 and has the highest overall mean, with the largest gains on fragmented support.

Critical strengths
  1. The tube genuinely generalizes prior trust regions (Σ=I\Sigma=I recovers RAMBO; fixed Σ\Sigma recovers ROMI), and one geometry drives both model learning and planning. The density gate fixes a real failure of Mahalanobis ellipsoids.

  2. Broad evaluation: 24 tasks, 5 seeds, 7 baselines in one codebase; ablations isolate the RVL loss and the anisotropic covariance; rollout NLL drops 39-54% versus a fixed horizon.

Critical weaknesses
  1. The link from "support is anisotropic" to "use a local covariance ellipsoid" is asserted, not derived. Model error is never shown to follow local Mahalanobis distance; Theorem 3.1 simply assumes a Gaussian density with covariance Σ∗\Sigma^*, turning the conclusion into a premise. Moreover, the neighbor covariance measures sampling geometry (and grows where data are sparse), so the largest tube appears where the model is least reliable.

  2. The theory does not support the claim: the "well-specified ensemble" assumption contradicts the paper's misspecification motivation, the bound compares trust regions under a fixed loss while the algorithm also changes the training objective, and terms (A)(A) and (A′)(A') are dimensionally incomparable, leaving the advantage on an unquantified constant. The trust score also mixes squared distance, variance, and standard deviation with fixed absolute thresholds, so "no task-specific tuning" is questionable.

Questions for authors
  1. State and test the missing assumption: why does local data covariance predict model-error anisotropy?
  2. Why shape the tube by data covariance but score trust by ensemble disagreement?
  3. How are the three terms of uu normalized, and do the fixed thresholds transfer across tasks?
  4. Give C1C_1--C4C_4 and make (A)(A), (A′)(A') comparable; is the claim robust to an optimally chosen ξ\xi?
  5. Are Table 1 returns raw, and how do you explain pen-E and hammer-E?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.