- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a flow matching policy formulation for offline goal-conditioned reinforcement learning as a construction upon the existing hierarchical methods. they derive a state-action ratio that weights low-level conditional flow matching and a state ratio that reweights expectile regression of the value, these are both integrated in low- and high-actor trainings. Authors report improved success rates on OGBench compared with the baselines.
Critical strengths
- the theory looks correct and evidential for all the parts of the method to be applied, including low-level flow objective which is related to occupancy ratio of the dual formulation acquired by the available reward. The state-action dual itself is the standard form of DICE, and there is no need to have a separate density estimator which provides potential intuition for better scaling
- the authors provide a reasonable amount of baselines to compare on a justified OGBench benchmark with multiple random seeds, additional ablation studies also demonstrate the necessity in each part of the main method to be integrated and justify the needed complexity of the idea
Critical weaknesses
- the empirical implementation of the method seems to be misaligned with the theoretical background (or at least it is not so straightforward to understand how exactly training of the method is implemented): is the key term to resolve corrections as on low-, so with high-level flows, but high-actor is being unweighted by this term. Here comes the question - is this fact addresses the ambiguity of the exact method pipeline or theoretical support is insufficient for this concrete hierarchical flow agent formulation?
- the main text attributes to weighting as the support for improved occupancy, thereby this fact should provide improved performance on stitch tasks on OGBench, but there is the gap with the baselines such as HIQL and OTA (e.g. 91±2 on antmaze-medium-stitch, against 94±1 for HIQL and 93±1 for OTA)
Questions for authors
- can you clarify the necessity of for high-level flow and provide a more motivational and intuitive aspects of how exactly the theoretical propositions reflect the empirical results?
- is the distillation really necessary for better success rate in tables 2-4? what is the performance without it and without Q-guidance?
- have you tried to isolate the effect of and on HIQL and OTA? what is more important - flow integration or these additional occupancy weights?