First, novelty and motivation are missing: The paper is the follow up of GFP, FAC and even FQL. But those works have a commun purpose: How to train, in the offline RL setup, an expression policy which is fast at inference and also avoid backpropagation through time. So, using on top of the one-step actor, the multi-step flow policy and an addition selections critics break the aim of fast inference. Apart from improving performance on benchmarks, it is unclear why BACI is relevant.
Second, the approach is too heavy. We first have to train GFP or FAC, then use the trained actors (one-step actor and multi-step BC) to train the decisions critics. At inference query both actor and the decisions critics on those two actions. On top of the GFP or FAC's hyperparameters, the method introduce 5 additionnals hyperparameters (t, p, r_\max, c_0, \lambda_d). Even though, (r_\max, c_0, \lambda_d) are fix to (0.15,1.10,0.25) for the main experiment, there is no justification or intuition about these chooses.
The writing style of the paper as well as its theoretical justification are difficult to follow and there are incorrect statement in the paper. For example: in row 101 and 102; the author wrote " za denotes the actor’s random input when the actor is stochastic. The flow policy starts from
an independent sample zF ". But the main idea of distillation behind GFP, FAC and FQL that z=zA=zF. This make me wonder if the authors have understood the methods there are suppose to build their work on. Even when, I follow the authors logic (zA and zF independant); how does πA depend on zF as stated in row 138? The overall framing of the paper is difficult to follow and some part are irrelevant ; e.g - line 043 'Existing methods improve how the policy or value function is learned from the offline data' (that is the definition of offline RL), - Assumption 1) " transition kernel is Markov" that is include into the definition of MDP; the assumption 1) is meaningless; line 158 and 159, Proposition 2, Theoreme 2, etc. Lets take Theoreme 2; for equation (9) to have any meaning; we first have assume the the assumption 1 (more than eleven statements);
then that
A=EK,s∼dπA[1d(s,K)≤rmax,aA=aF1minj∈1,2qψj(s,aF)−minj∈1,2qψj(s,aA)>c0+λdd(s,K)(QπA(s,aF)−QπA(s,aA)−c0−λdd(s,K))]>0,
(EK,s∼dπA[1d(s,K)≤rmax,aA=aF1minj∈1,2qψj(s,aF)−minj∈1,2qψj(s,aA)>c0+λdd(s,K)])×(supsEK[1d(s,K)≤rmax,aA=aF1minj∈1,2qψj(s,aF)−minj∈1,2qψj(s,aA)>c0+λdd(s,K)(QπA(s,aF)−QπA(s,aA)−c0−λdd(s,K))]−infsEK[1d(s,K)≤rmax,aA=aF1minj∈1,2qψj(s,aF)−minj∈1,2qψj(s,aA)>c0+λdd(s,K)(QπA(s,aF)−QπA(s,aA)−c0−λdd(s,K))])>0
and
0<p<(EK,s∼dπA[1d(s,K)≤rmax,aA=aF1minj∈1,2qψj(s,aF)−minj∈1,2qψj(s,aA)>c0+λdd(s,K)])×(supsEK[1d(s,K)≤rmax,aA=aF1minj∈1,2qψj(s,aF)−minj∈1,2qψj(s,aA)>c0+λdd(s,K)(QπA(s,aF)−QπA(s,aA)−c0−λdd(s,K))]−infsEK[1d(s,K)≤rmax,aA=aF1minj∈1,2qψj(s,aF)−minj∈1,2qψj(s,aA)>c0+λdd(s,K)(QπA(s,aF)−QπA(s,aA)−c0−λdd(s,K))])(1−y)EK,s∼dπA[1d(s,K)≤rmax,aA=aF1minj∈1,2qψj(s,aF)−minj∈1,2qψj(s,aA)>c0+λdd(s,K)(QπA(s,aF)−QπA(s,aA)−c0−λdd(s,K))]
No ablation on hyperparameters introduce (r_\max, c_0, \lambda_d), insuffisant related works.