- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes ACH, an offline-to-online RL algorithm that adaptively changes the action chunking size. It learns a causal Transformer-based value function, and adaptively selects the action chunk size from the Q values. ACH consistently outperforms prior offline-to-online works, including OGBench and SimplerEnv (with VLAs).
Critical strengths
S1. Method is straightforward and simple.
- The method trains a Transformer value function and extracts the policy with DDPG + BC (from flow BC policy).
- Action chunk size is adaptively selected via Boltzman soft policy.
- This can be seamlessly applied to pre-trained policies such as VLAs.
S2. Strong experimental results.
- ACH consistently outperforms prior offline-to-online methods in OGBench and SimplerEnv with VLAs.
- Also, ablation studies and qualitative anlaysis (Section 4.2) support that adaptive action chunking is important and ACH's objective improves the policy via adaptive action chunking.
Critical weaknesses
W1. Lack of theoretical understanding of the objectives
- The policy is decomposed to two parts, action chunking policy and size policy , where is a boltzman policy.
- While individual design choices are reasonable, the paper does not contain where the policy and Q function will converge from these objectives.
- For example, if there is no BC constraint, the algorithm will converge to a markovian policy without action chunking, as markovian policy is always as good as nonmarkovian policy due to reactivity.
- It will be great if the authors can provide what policy / Q function is going to be learned by this methodology.
Questions for authors
Q1. Theoretical understanding of the objective
- Would it be possible to provide explanation or intuition on what policy / Q function is going to be learned by this methodology?
- This will help the readers on understanding the objective of the algorithm.
6 comments
Thank you for the clarification and providing the theoretical analysis.
However, I think the proposed theoretical result does not address my concern on "what policy / Q function is going to be learned by this methodology", as 1) this result does not incorporate BC constraint, 2) therefore this result again shows that it will converge to a markovian optimal policy (or action chunked policy that is equivalent to markovian policy).
I'll maintain the positive score.
Thank you for the quick response. We appeciate your positive evaluation.
As noted, when the behavior regularization term is not present in Eq. (8), then Eqs. (7) and (8) form a case of continuous version of policy iteration for semi-MDP. However, with the behavior regularization term, analysis is difficult. Especially, the regularization term is in a soft form which makes analysis more difficult. We conjecture that if we change the soft behavior constraint to a hard constraint as then the constrained policy iteration may converge to a policy maximizing Q under the behavior regularization. It seems that this conjecture requires further in-depth analysis. If the result is available, we will get back to you again.
Thank you.
Authors
Authors' response to reviewer PNwf
We thank the reviewer for the thoughtful review. The question of what the objectives of ACH learn motivated a new theoretical analysis, which we added as Appendix B.1 and summarized in Section 3.4 of the revised manuscript (changes are marked in blue). We summarize the results below.
R3-1. On the theoretical understanding of the objectives (W1, Q1)
The overall procedure of ACH is captured by the composite policy μ=(πθ,πL), which at each decision state samples an action chunk from πθ and an execution length from πL, executes the corresponding prefix open-loop, and makes its next decision at the reached state.
Since an action chunk is a semi-Markov option (Sutton et al., 1999), learning with adaptive chunks forms a Semi-MDP (SMDP), and standard SMDP theory can be adapted to our setting. Assuming exact policy evaluation and improvement in the tabular setting with deterministic transitions, we show the following:
What the Q function learns (Proposition 1, policy evaluation). For a fixed μ, the value functions satisfy
Qμ(st,at:t+n)=Rtn+γn+1Vμ(st+n+1),Vμ(st)=Eat:t+h′∼πθ, ℓ∼πL[Qμ(st,at:t+ℓ′)],
which are the SMDP Bellman expectation equations of μ and have a unique solution (Qμ,Vμ). The two targets of the mean-square loss in Eq. (7) correspond to these two equations. In other words, the critic of ACH learns the value of every prefix under the current chunk policy and the current length policy.
How the policy is improved (Proposition 2, policy improvement). If a new composite policy μ′ satisfies Eμ′[Qμ(st,at:t+ℓ′)]≥Vμ(st) at every state, then Vμ′≥Vμ. The improvement condition holds (i) when the chunk policy is kept and the length is selected greedily, which is the selection used at evaluation, and (ii) when the chunk policy additionally generates a chunk that maximizes the best prefix value. The value term in Eq. (8) similarly encourages the actor to generate chunks with higher expected prefix values, while Proposition 2 specifies the condition under which an update guarantees improvement.
Where the values converge (Theorem 1, policy iteration). Alternating exact evaluation of Proposition 1 and globally greedy improvement of Proposition 2 (ii) gives Vμ0≤Vμ1≤⋯≤V∗ and Vμk→V∗, where V∗ is the optimal value function of the MDP. Thus, the value function converges to the optimal value function of the original MDP under this exact policy iteration.
Optimal chunks (Theorem 2). There exists an optimal composite policy μ∗ whose chunk attains V∗ at every prefix length. Hence, restricting the policy to open-loop chunks with adaptive lengths does not lose optimality.
Chunking and the optimal Markovian policy. A policy with action chunking cannot be better than the optimal Markovian policy, which selects every action from the current state. Under the exact policy iteration of Theorem 1, the values converge to the value of the optimal Markovian policy. By Theorem 2, attaining the optimal value does not require a chunk length of one. However, the chunk length matters for stochastic and suboptimal policies encountered during learning, and in particular for the behavior-regularized policy of Eq. (8). For any such μ, the values of adjacent lengths satisfy
Qμ(st,at:t+n)−Qμ(st,at:t+n−1)=γn(Qμ(st+n,at+n)−Vμ(st+n)).
Extending a chunk by one step increases the value exactly when the pre-planned action at+n is better than replanning at st+n, i.e., drawing a new chunk and a new length. When the reached state is consistent with the plan, replanning can introduce the sampling noise of the policy without improving the action, and longer chunks are favored. When the pre-planned action, which is generated from st alone, no longer suits the reached state, shorter chunks are favored. Comparing the values of all prefixes lets ACH balance these two cases at each state, which is consistent with its improved sample efficiency during learning.
The practical algorithm. In Eq. (8), the BC term keeps the actor close to the behavior policy, and during online interaction the soft πL provides exploration over chunk lengths. Theorem 1 assumes exact evaluation and globally greedy improvement, whereas the practical algorithm uses function approximation and gradient updates of the behavior-regularized actor. Removing the BC term alone does not make these updates equivalent to the policy iteration of Theorem 1. We state the scope of the analysis at the end of Appendix B.1.
We thank the reviewer again for the constructive suggestion and would be happy to provide any further clarification.