- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces Actor-Critic with Action Chunking (AC2), an RL method for LLM reasoning that uses a learned critic to assign credit to action chunks rather than relying on terminal rewards, allowing policy updates without rolling every trajectory to completion. To make critic-based credit reliable, AC2 uses local readiness, reference solutions in the critic’s context, and action chunking instead of token-level advantages. On a math proof benchmark, AC2 outperforms GRPO while using less decoding compute and fewer training steps, and ablations show all three design choices matter. The central claim is that LLM RL can trust a learned critic far more than current methods assume, enabling more efficient trajectory reuse and off-policy learning.
Critical strengths
-
Challenges a core assumption in LLM RL. The paper argues and empirically supports that a learned critic can be trusted far more than current methods assume, offering a concrete recipe (local readiness, reference solutions, action chunking) to make critic-based credit assignment practical instead of discarding the critic in favor of terminal rewards or Monte Carlo rollouts. This is a promising direction.
-
Simple yet elegant design. AC2 achieves finer-grained credit assignment through a minimal set of design choices: local readiness, reference solutions, and action chunking without introducing a complicated new framework, making the method easy to understand, implement, and build upon.
Critical weaknesses
-
Contribution is incremental: The main contribution should be further justified, since prefix-branching and group-relative credit assignment are not new (e.g., Vine-PPO). AC2’s novelty is the reliable-critic recipe: local readiness, reference solutions, and action chunking which enables chunk-level credit without terminal rewards and outperforms GRPO with less compute, with ablations confirming each component matters. To fully justify the contribution, the paper should directly compare against Vine-PPO and other prefix-branching baselines and show the gains generalize beyond one model and benchmark.
-
Random Prefix Boundary: A key weakness is the unjustified reliance on random prefix boundaries for action chunking: cuts can land mid-step or mid-expression, producing incoherent prefixes that make critic values noisy, create distribution shift, and yield variable chunk lengths that bias credit. Local readiness only checks problem-level, not prefix-level, reliability, and reference solutions may misalign with broken prefixes.
Questions for authors
- Prefix selection: Why random cuts? Have you tried semantic/step boundaries or a learned cut policy, and how much does it matter?
- Chunk size: Why choose around 10k tokens? How sensitive is performance to chunk length, and does the optimal size depend on task horizon? Can you provide an ablation study to sweep a wider range of chunk size, i.e., 500, 1k, 5k ....?
- Generalization: Does the gain hold on other models, domains (code, agentic tasks), and longer horizons, or only Qwen3-4B on IMO-ProofBench?
4 comments
Thank you again for your feedback. Have we addressed your concerns in our response or is there anything else we could clarify?
Thank you for providing new results with Qwen model. The new Qwen3-1.7B + CodeIO experiment with 4 seeds and a direct VinePPO comparison is exactly the kind of evidence that was missing. It addresses “one model, one benchmark” and “no direct prefix-branching baseline.”
What remains only partially / not addressed:
-
Random prefix cuts: The authors argue randomness is a feature because it avoids defining “steps,” and that the critic is trained on the same cut distribution with low error. That helps, but it does not show that training on semantically incoherent prefixes yields better policy updates than semantic/step boundaries. A low critic error at random cuts is necessary but not sufficient. A random-vs-semantic-cut ablation would be much more convincing.
-
Contribution justification. The response does not address the concern that the paper’s contribution needs clearer positioning relative to prior prefix-branching / Monte Carlo methods. The new VinePPO comparison and the new model/domain experiment help empirically, but they do not by themselves justify what is conceptually new. In fact, your note that Prefix-GRPO is “very similar to Vine-PPO” and that AC2 mainly removes the need for every continuation to reach a terminal reward makes this positioning more important.
-
Unaddressed questions: reference-solution realism, local-readiness threshold sensitivity, full compute accounting including critic training/replay are not really answered in this response.
Without clearing these concerns, this reviewer tends to maintain the original judgement.
Thank you for the follow-up. We’re glad the four-seed CodeIO experiment and the direct VinePPO comparison addressed your concerns about generality and the missing baseline. We’d like to respond to your remaining concerns
Random versus semantic cuts. We use random cuts to avoid requiring a task-specific definition of a reasoning step. The low critic error on this distribution supports using the critic at these cut points. We agree that semantic cuts could further improve AC2, but given that AC2 already outperforms GRPO with random cuts, we view a direct comparison with semantic cuts as a promising direction for future work rather than a prerequisite for our current conclusions.
Contribution justification. Our method differs from prefix-branching based method such as VinePPO significantly given the use of the learned critic. As you mentioned, the main benefit of AC2 is "removes the need for every continuation to reach a terminal reward".
Remaining Questions. We unfortunately could not map the keywords to your original review. Based on our best understanding, here is our response:
-
Reference-solution realism: We assume this is asking whether it is realistic to assume we have reference solution. One of the main feature of our algorithm is that all the correct solution are collected during the RL process, and we don't need a separate oracle to provide the reference solution.
-
local-readiness threshold sensitivity: We assume this is asking whether our algorithm is robust to the error threshold of local readiness. We found the same local readiness threshold works across the two settings we tested.
-
full compute accounting including critic training/replay: We assume this is asking whether AC2 provides real GPU hours saving beyond decoding flops. During rebuttal, we add Figure 6 to the paper that shows Decoding FLOPs correlates closely with GPU hours across methods.
Also for our new experiments on CodeIO, as each run uses the same resources (a single 8xH100 node), we are able to use the ideal H100 hours metric exactly. Figure 5 of the updated PDF shows that for CodeIO AC2 outperforms other methods when measuring compute with this metric directly.
Thank you for your feedback! We include our responses below:
We run new experiments with a new model on a new domain, comparing directly to VinePPO, across 4 seeds. Specifically, we train Qwen3-1.7B on CodeIO, a code reasoning task where the model is given a program and predicts its input or output. We compare AC2 with GRPO, PPO and VinePPO, running four seeds per method. Every run uses one node of 8 H100 GPUs, so we compare methods directly in H100 hours. AC2 reaches the highest seed-mean peak validation accuracy, 43.3% against 41.7% for GRPO, 40.0% for PPO and 40.5% for VinePPO. VinePPO is also far more expensive per step, as it samples roughly 10× more tokens per step than GRPO to estimate values from Monte Carlo continuations. The results are in Figure 5 of the new Appendix A (Robustness across model, task, and seed) in the updated PDF.
Also we note that prefix-GRPO is a very similar to Vine-PPO and so it is not too surprising AC2 also beats Vine-PPO. Both estimate a prefix's value by averaging the terminal rewards of full continuations sampled from it, and both use this as the baseline for each continuation's advantage. The main difference is that VinePPO branches at many points along a trajectory, while Prefix GRPO branches at one prefix per group. Both still need every continuation to reach a terminal reward, which is the cost AC2 removes.
Action 1: We add the CodeIO experiments (Qwen3-1.7B, 4 seeds), including a VinePPO baseline, to Figure 5 of the new Appendix A in the updated PDF.
Thank you for raising this. We actually think random prefix is a feature of the algorithm, as it means we do not have to define any notion of “step” that might change for different problems or domains. We note the cut points are the same distribution the critic is trained on. Figure 2 shows its error at these cuts is low and falls over training. You also said “and yield variable chunk lengths that bias credit” however this is not possible. The critic always judges a full 10k chunk, unless the rollout ends in which case we use terminal reward.
10k, or rather 20% of the full context, was chosen somewhat heuristically to be a “meaningful” chunk of work. We provide provide 2 chunk size ablations in the paper. Full 50k chunk is simply prefix GRPO, which we beat. We also provide 2k chunk size ablation in Figure 3 right which performs worse than our 10k run. Our intuition here is that you need some balance between large enough chunks such that the true value of the state at the start and end of the chunk is different, but small enough such that you can save compute per step by rolling out a fewer tokens than a full trajectory requires.
Summary
Thank you for the detailed review! In response we have made several changes that we think improve the paper, most notably the new VinePPO comparison and the results on a new model and domain. Do you feel these address your concerns enough to warrant a higher score? If not, please let us know if you have any more questions.