- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses offline goal-conditioned RL. The idea is to learn a "forcaster" which rolls out future (embeddings of) observations, a "refiner" that iteratives on imagined actions, and a policy which adjudicates the next action on the basis of these model's outputs, but which interestingly is not constrained to pick the utility-maxing imagined rollout.
They evaluate on OGBench, offline goal conditioned simulated robot manipulation tasks. They compare against published OGBench results, a JEPA baseline and a TD-MPC baseline (which admittedly I'm not familiar with), and various ablations/versions of their approach, finding their approach generally >= alternatives, and that it improves with extra refinement steps.
Critical strengths
The paper blends a bunch of nice research areas -- offline RL, test time compute, world models -- packaging them in one system. It's nice that the system does not degrade and in fact can somewhat improve with extra test time compute.
Critical weaknesses
- The idea that extra test time compute can aid model based control is an old one. It's great your method benefits from extra inference time compute, but so does a planner. How does your method's test time compute scale vs. a planner? No analysis of runtime, which is odd for a work whose signature contribute is advertised in the title as "test time compute". Can you make graphs showing how your scaling fares vs. conventional MPC / planning? Can you compare to MuZero / other systems that integrate sophisticated search (not CEM) with learning?
- Complex method -- I get that you're combining many cool ideas at once, but the payoff is not commensurate with the complexity. There's 5 different losses, two different training stages, and arbitrary hyperparameters with no corresponding sensitivity analysis.
- Little discussion of nondeterminism and partial observability -- these are central concerns in model based control and deeply influence the nature of the planning problem, and indeed what we expect the forcaster to represent. Is OGBench stochastic? If so we expect the embeddings to encode belief states, not future observations. If your method effectively crumbles under nondeterminism, that would be good to be upfront about.
Questions for authors
How does your method's use of test time compute scale differently/the same/wall-clock-wise vs planning approaches? Can you handle stochastic environments, not just in practice on-average on a benchmark, but theoretically in-principle?