acceptodds
Under review as a conference paper at ICLR 2027

Trust the Critic More

Abstract

Standard language model RL algorithms credit every token of a long rollout with the same advantage determined by the terminal reward. Actor–critic methods can provide finer-grained credit assignment, but learned critics are generally considered too inaccurate to trust for RL for LLMs. In recent works, even when a critic is present, it is used only for baseline estimation, so every trajectory must be rolled out to its terminal reward (Venkatraman et al., 2026; Pan et al., 2026). We introduce Actor-Critic with Action Chunking (AC2) that removes the need to roll every trajectory to completion. AC2 instead assigns credit to action chunks: short continuations of prefixes of past trajectories. A learned critic scores the state reached at the end of each action chunk, allowing the policy to update without observing a terminal reward. We make critic-based credit assignment reliable through three design choices. First, we introduce local readiness, which uses critic-based updates on a problem only when the critic is sufficiently accurate on that particular problem. Second, when available, we provide the critic with a reference solution from a previous successful rollout. Third, we assign credit over action chunks of 10k tokens rather than individual tokens, giving the critic a more meaningful amount of work to evaluate. We train Qwen3-4B on FineProofs-RL using AC2 and evaluate on IMO-ProofBench. AC2 exceeds GRPO's peak validation score of using fewer decoding FLOPs. This gain comes from two sources: (1) AC2 requires fewer training steps to reach this score, and (2) each step generates fewer tokens because the policy does not need to continue every trajectory to completion. Conceptually, we demonstrate that we can remove the need to roll out every trajectory to completion, opening up a large previously unexplored design space for LLM RL algorithms.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, actor critic, RLVR

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8fGcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Actor-Critic with Action Chunking (AC2), an RL method for LLM reasoning that uses a learned critic to assign credit to action chunks rather than relying on terminal rewards, allowing policy updates without rolling every trajectory to completion. To make critic-based credit reliable, AC2 uses local readiness, reference solutions in the critic’s context, and action chunking instead of token-level advantages. On a math proof benchmark, AC2 outperforms GRPO while using less decoding compute and fewer training steps, and ablations show all three design choices matter. The central claim is that LLM RL can trust a learned critic far more than current methods assume, enabling more efficient trajectory reuse and off-policy learning.

Critical strengths
  1. Challenges a core assumption in LLM RL. The paper argues and empirically supports that a learned critic can be trusted far more than current methods assume, offering a concrete recipe (local readiness, reference solutions, action chunking) to make critic-based credit assignment practical instead of discarding the critic in favor of terminal rewards or Monte Carlo rollouts. This is a promising direction.

  2. Simple yet elegant design. AC2 achieves finer-grained credit assignment through a minimal set of design choices: local readiness, reference solutions, and action chunking without introducing a complicated new framework, making the method easy to understand, implement, and build upon.

Critical weaknesses
  1. Contribution is incremental: The main contribution should be further justified, since prefix-branching and group-relative credit assignment are not new (e.g., Vine-PPO). AC2’s novelty is the reliable-critic recipe: local readiness, reference solutions, and action chunking which enables chunk-level credit without terminal rewards and outperforms GRPO with less compute, with ablations confirming each component matters. To fully justify the contribution, the paper should directly compare against Vine-PPO and other prefix-branching baselines and show the gains generalize beyond one model and benchmark.

  2. Random Prefix Boundary: A key weakness is the unjustified reliance on random prefix boundaries for action chunking: cuts can land mid-step or mid-expression, producing incoherent prefixes that make critic values noisy, create distribution shift, and yield variable chunk lengths that bias credit. Local readiness only checks problem-level, not prefix-level, reliability, and reference solutions may misalign with broken prefixes.

Questions for authors
  1. Prefix selection: Why random cuts? Have you tried semantic/step boundaries or a learned cut policy, and how much does it matter?
  2. Chunk size: Why choose around 10k tokens? How sensitive is performance to chunk length, and does the optimal size depend on task horizon? Can you provide an ablation study to sweep a wider range of chunk size, i.e., 500, 1k, 5k ....?
  3. Generalization: Does the gain hold on other models, domains (code, agentic tasks), and longer horizons, or only Qwen3-4B on IMO-ProofBench?
4 comments

Thank you for your feedback! We include our responses below:

Generalization: Does the gain hold on other models, domains

and show the gains generalize beyond one model and benchmark.

To fully justify the contribution, the paper should directly compare against Vine-PPO and other prefix-branching baselines

We run new experiments with a new model on a new domain, comparing directly to VinePPO, across 4 seeds. Specifically, we train Qwen3-1.7B on CodeIO, a code reasoning task where the model is given a program and predicts its input or output. We compare AC2 with GRPO, PPO and VinePPO, running four seeds per method. Every run uses one node of 8 H100 GPUs, so we compare methods directly in H100 hours. AC2 reaches the highest seed-mean peak validation accuracy, 43.3% against 41.7% for GRPO, 40.0% for PPO and 40.5% for VinePPO. VinePPO is also far more expensive per step, as it samples roughly 10× more tokens per step than GRPO to estimate values from Monte Carlo continuations. The results are in Figure 5 of the new Appendix A (Robustness across model, task, and seed) in the updated PDF.

Also we note that prefix-GRPO is a very similar to Vine-PPO and so it is not too surprising AC2 also beats Vine-PPO. Both estimate a prefix's value by averaging the terminal rewards of full continuations sampled from it, and both use this as the baseline for each continuation's advantage. The main difference is that VinePPO branches at many points along a trajectory, while Prefix GRPO branches at one prefix per group. Both still need every continuation to reach a terminal reward, which is the cost AC2 removes.

Action 1: We add the CodeIO experiments (Qwen3-1.7B, 4 seeds), including a VinePPO baseline, to Figure 5 of the new Appendix A in the updated PDF.

A key weakness is the unjustified reliance on random prefix boundaries for action chunking: cuts can land mid-step or mid-expression, producing incoherent prefixes that make critic values noisy

Prefix selection: Why random cuts? Have you tried semantic/step boundaries or a learned cut policy, and how much does it matter?

Thank you for raising this. We actually think random prefix is a feature of the algorithm, as it means we do not have to define any notion of “step” that might change for different problems or domains. We note the cut points are the same distribution the critic is trained on. Figure 2 shows its error at these cuts is low and falls over training. You also said “and yield variable chunk lengths that bias credit” however this is not possible. The critic always judges a full 10k chunk, unless the rollout ends in which case we use terminal reward.

Chunk size: Why choose around 10k tokens? How sensitive is performance to chunk length, and does the optimal size depend on task horizon? Can you provide an ablation study to sweep a wider range of chunk size, i.e., 500, 1k, 5k ....?

10k, or rather 20% of the full context, was chosen somewhat heuristically to be a “meaningful” chunk of work. We provide provide 2 chunk size ablations in the paper. Full 50k chunk is simply prefix GRPO, which we beat. We also provide 2k chunk size ablation in Figure 3 right which performs worse than our 10k run. Our intuition here is that you need some balance between large enough chunks such that the true value of the state at the start and end of the chunk is different, but small enough such that you can save compute per step by rolling out a fewer tokens than a full trajectory requires.

Summary

Thank you for the detailed review! In response we have made several changes that we think improve the paper, most notably the new VinePPO comparison and the results on a new model and domain. Do you feel these address your concerns enough to warrant a higher score? If not, please let us know if you have any more questions.

Thank you again for your feedback. Have we addressed your concerns in our response or is there anything else we could clarify?

Thank you for providing new results with Qwen model. The new Qwen3-1.7B + CodeIO experiment with 4 seeds and a direct VinePPO comparison is exactly the kind of evidence that was missing. It addresses “one model, one benchmark” and “no direct prefix-branching baseline.”

What remains only partially / not addressed:

  • Random prefix cuts: The authors argue randomness is a feature because it avoids defining “steps,” and that the critic is trained on the same cut distribution with low error. That helps, but it does not show that training on semantically incoherent prefixes yields better policy updates than semantic/step boundaries. A low critic error at random cuts is necessary but not sufficient. A random-vs-semantic-cut ablation would be much more convincing.

  • Contribution justification. The response does not address the concern that the paper’s contribution needs clearer positioning relative to prior prefix-branching / Monte Carlo methods. The new VinePPO comparison and the new model/domain experiment help empirically, but they do not by themselves justify what is conceptually new. In fact, your note that Prefix-GRPO is “very similar to Vine-PPO” and that AC2 mainly removes the need for every continuation to reach a terminal reward makes this positioning more important.

  • Unaddressed questions: reference-solution realism, local-readiness threshold sensitivity, full compute accounting including critic training/replay are not really answered in this response.

Without clearing these concerns, this reviewer tends to maintain the original judgement.

Thank you for the follow-up. We’re glad the four-seed CodeIO experiment and the direct VinePPO comparison addressed your concerns about generality and the missing baseline. We’d like to respond to your remaining concerns

Random versus semantic cuts. We use random cuts to avoid requiring a task-specific definition of a reasoning step. The low critic error on this distribution supports using the critic at these cut points. We agree that semantic cuts could further improve AC2, but given that AC2 already outperforms GRPO with random cuts, we view a direct comparison with semantic cuts as a promising direction for future work rather than a prerequisite for our current conclusions.

Contribution justification. Our method differs from prefix-branching based method such as VinePPO significantly given the use of the learned critic. As you mentioned, the main benefit of AC2 is "removes the need for every continuation to reach a terminal reward".

Remaining Questions. We unfortunately could not map the keywords to your original review. Based on our best understanding, here is our response:

  1. Reference-solution realism: We assume this is asking whether it is realistic to assume we have reference solution. One of the main feature of our algorithm is that all the correct solution are collected during the RL process, and we don't need a separate oracle to provide the reference solution.

  2. local-readiness threshold sensitivity: We assume this is asking whether our algorithm is robust to the error threshold of local readiness. We found the same local readiness threshold works across the two settings we tested.

  3. full compute accounting including critic training/replay: We assume this is asking whether AC2 provides real GPU hours saving beyond decoding flops. During rebuttal, we add Figure 6 to the paper that shows Decoding FLOPs correlates closely with GPU hours across methods.

Also for our new experiments on CodeIO, as each run uses the same resources (a single 8xH100 node), we are able to use the ideal H100 hours metric exactly. Figure 5 of the updated PDF shows that for CodeIO AC2 outperforms other methods when measuring compute with this metric directly.

Reviewer_NFQzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces a new framework of actor-critic using action chunking. The method keeps a replay buffer with complete rollouts and each time it samples both prefixes and fresh problems from the buffer and the problem set without replacement. If a problem satisfies a readiness score then it will be put into D_{ready}. If a prefix of a question is not ready, then the LM will be asked to sample until termination. Otherwise, it will be asked to sample a chunk of length b. Then, advantages are computed using a group relative advantage estimation similar to GRPO. (for problems that are not ready the rewards are replaced with values). Empirical results show that the method is effective and efficient. Analysis also justifies and explains the components of the method.

Critical strengths
  1. The new actor-critic framework is novel and practical. It's easy to understand and follow and it shows great potential for future study.
  2. Empirical results are strong and the analysis is comprehensive.
Critical weaknesses
  1. Only one model and a single dataset is used.
  2. Only two critic-free method is compared. How does it compare to other actor-critic methods like PPO?
Questions for authors
  1. What is the performance like when training with more compute? Is it saturated?
2 comments

Thank you for your feedback! We include our responses below:

Only one model and a single dataset is used.

We run new experiments with a new model on a new domain, across 4 seeds. Specifically, we train Qwen3-1.7B on CodeIO, a code reasoning task where the model is given a program and predicts its input or output. We compare AC2 with GRPO, PPO and VinePPO, running four seeds per method. AC2 reaches the highest seed-mean peak validation accuracy, 43.3% against 41.7% for GRPO, 40.0% for PPO and 40.5% for VinePPO. GRPO and PPO decline after their peaks while AC2 keeps improving. The results are in Figure 5 of the new Appendix A (Robustness across model, task, and seed) in the updated PDF.

Action 1: We add the CodeIO experiments (Qwen3-1.7B, 4 seeds) to Figure 5 of the new Appendix A in the updated PDF.

Only two critic-free method is compared. How does it compare to other actor-critic methods like PPO?

In the CodeIO setting we also run PPO. Our PPO uses a separate learned critic of the same size as the policy (Qwen3-1.7B) with GAE (γ = λ = 1). We find AC2 outperforms PPO, reaching 43.3% seed-mean peak validation accuracy against 40.0%, and PPO declines after its peak while AC2 keeps improving with more compute. We also compare with VinePPO, which replaces the learned critic with Monte Carlo value estimates, and AC2 outperforms it as well (40.5%). See Figure 5 of the new Appendix A in the updated PDF.

Action 2: We add a PPO baseline on CodeIO to Figure 5 of the new Appendix A in the updated PDF.

What is the performance like when training with more compute? Is it saturated?

We did not run for longer than shown in Figure 1, but looking at the results there we think the method has plateaued in compute. AC2 reaches a higher plateau than GRPO suggesting it is more data efficient. To keep seeing more gains with compute we would want to expand dataset of training problems. For the CodIO setting (Figure 5), AC2 improves with more compute while GRPO and PPO reach a peak and then decrease in performance.

Summary

Thank you again for your review! We believe the changes made in response, in particular the new model, domain and PPO baseline, strengthen the paper. Do these changes warrant a score increase? If not, please let us know if you have any other questions.

Thank you again for your feedback. Have we addressed your concerns in our response or is there anything else we could clarify?

Reviewer_BAkROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Actor-Critic with Action Chunking (AC2), which trains language models using short continuations from replayed prefixes. When a problem satisfies a critic-readiness criterion, predicted endpoint values replace terminal rewards for some policy updates. The method combines problem-specific readiness, reference-conditioned value prediction, and 10,000-token chunks. On Qwen3-4B trained on FineProofs-RL, AC2 exceeds GRPO’s peak IMO-ProofBench mean score using approximately 2.5× fewer policy-decoding FLOPs and achieves a higher peak score, 20.57% versus 18.50%.

Critical strengths

(1) Relevant problem and potentially significant efficiency gains. Avoiding complete rollouts is valuable for long reasoning tasks. The paper offers a concrete demonstration that learned critics can support updates from unfinished continuations.

(2) Thoughtful algorithmic design. Local readiness addresses heterogeneous critic reliability across problems. Reference conditioning, periodic terminal-reward audits, and replay provide practical mechanisms for supporting bootstrapped learning.

(3) Useful empirical diagnostics. Prefix GRPO helps assess whether replay alone explains the gains. The reference-conditioning analysis and comparisons between predicted and terminal-reward advantages provide evidence beyond aggregate benchmark scores. The appendix candidly documents several experimental limitations and implementation details.

Critical weaknesses

(1) Decoding FLOPs exclude prefill, critic inference, judging, and parameter updates. Correlation with GPU-hours within AC2 does not establish comparable overhead across algorithms. The reported 2.5× result supports a policy-decoding saving, not yet an equivalent end-to-end training saving. (2) Results cover one model, one training domain, and 60 evaluation problems, without reported variation across training seeds or uncertainty for the main comparison. The local-readiness comparison uses different training histories; the chunk-size experiment also changes prefix-cut spacing. These limitations, acknowledged in the appendix, weaken claims that individual components are essential. Comparisons against a strong learned-critic baseline are also missing.

Questions for authors

(1) What end-to-end compute saving remains after including all training overhead? (2) Can matched-history ablations isolate local readiness and chunk length? (3) How often do eligible problems subsequently fail terminal-reward audits, and why is eligibility never revoked? (4) Can repeated runs and problem-level confidence intervals establish the score improvement? (5) How does AC2 compare with truncated positive-λ\lambda GAE under matched compute?

2 comments

Thank you for your feedback! We provide our response below:

Results cover one model, one training domain

Comparisons against a strong learned-critic baseline are also missing

(4) Can repeated runs and problem-level confidence intervals establish the score improvement?

Without reported variation across training seeds

We run new experiments with a new model on a new domain comparing to learnt-critic baseline PPO, and do so across 4 seeds. We demonstrate that AC2 outperforms baselines on a new domain, with a new model, and our main result is not an artifact of a single seed.

Specifically, we train Qwen3-1.7B on CodeIO, a code reasoning task where the model is given a program and predicts its input or output. We compare AC2 with GRPO, PPO (learned-critic baseline) and VinePPO, running four seeds per method. Every run uses one node of 8 H100 GPUs, so here we can compare methods directly in H100 hours rather than decoding FLOPs. AC2 reaches the highest seed-mean peak validation accuracy, 43.3% against 41.7% for GRPO, 40.0% for PPO and 40.5% for VinePPO. GRPO and PPO decline after their peaks while AC2 keeps improving. The results are in Figure 5 of the new Appendix A (Robustness across model, task, and seed) in the updated PDF.

Overall we spend 7,800 H100 hours on these new experiments. Given compute constraints it is infeasible for us to run rerun our main results over 4 seeds. However, Section 4.2 presents three variants of AC2, each an independent training run with its own sampled rollouts, that all learn faster than GRPO. While these are not seed replicates, they are support evidence that our main AC2 results are not an artifact of a single lucky run.

Action 1: We run new experiments showing AC2 is effective in a new domain, with a new model, and outperforms critic-based baselines. We are confident in the result as we run for 4 seeds, as the reviewer requested. The results are in Figure 5 of the new Appendix A in the updated PDF.

Decoding FLOPs exclude prefill, critic inference, judging, and parameter updates. Correlation with GPU-hours within AC2 does not establish comparable overhead across algorithms.

What end-to-end compute saving remains after including all training overhead?

Thank you for raising this. We believe the true ideal metric is something like “H100 hours” spent. However we cannot compare our main results using this metric as we use different GPU types for different runs (due to availability). As you say, from Figure 22 we see that Decoding FLOPs correlates very well with this ideal metric. It was a flaw to not provide evidence that Decoding FLOPs also correlates with the ideal runtime our baselines. We add Figure 6 to the paper that shows for the baseline methods Decoding flops also correlates very closes with GPU hours.

Finally for our new experiments on CodeIO, as each run uses the same resources (a single 8xH100 node), we are able to use the ideal H100 hours metric exactly. Figure 5 of the updated PDF shows that for CodeIO AC2 still outperforms other methods when measuring compute with this metric.

Action 2: We add Figure 6 to the paper that shows Decoding FLOPs correlates closely with GPU hours across methods.

(2) Can matched-history ablations isolate local readiness and chunk length?

For chunk length, we think the from scratch comparison in the paper is the correct one to run. Chunk length determines which states the critic is trained on from step 1, so a fork from a 10k checkpoint would inherit a critic trained only for 10k chunks. The local-readiness ablation is already a fork, from AC2 at step 20. A fully matched branch, differing only in local readiness, would be ideal, but we prioritized rebuttal compute for new domain results.

(3) How often do eligible problems subsequently fail terminal-reward audits, and why is eligibility never revoked?

In the main run, 19.4% of audited groups exceed τlocal\tau_{\text{local}}, and this rate stays roughly constant over training (17–21%). Most audits are accurate, with 54% having zero error. The newly added Figure 21 of the paper shows the distribution of audited group errors. Revoking readiness after repeated audit failures is a natural extension we have not tested.

Action 3: We add Figure 21 and text in Appendix B.4 discussing the audit error and the possiblity of revoking local readiness, this is referenced in the main body on page 7.

How does AC2 compare with truncated positive-(\lambda) GAE under matched compute?

The new PPO baseline on CodeIO is a lambda=1 GAE estimator, and AC2 outperforms it. If we have misunderstood question though please let us know and we would be happy to run something else!

Summary

Thank you for your feedback! We have made a number of changes to the paper in response that we think improve the paper. Do you think this warrants a score increase? If not please let us know if you have any remaining questions we could help resolve.

Thank you again for your feedback. Have we addressed your concerns in our response or is there anything else we could clarify?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Thank you for your thoughtful feedback! The main shared criticism from all reviewers was that our results cover a single model, dataset, and seed, and we should test against a PPO and VinePPO baseline. We agree and in response to this conducted new experiments showing AC2’s performance improvement is robust across a new model and setting with multiple seeds. Overall we spend 7,800 H100 hours on these new experiments.

Specifically, we train Qwen3-1.7B on CodeIO, a code reasoning task where the model predicts a program's input or output, and compare AC2 with GRPO, PPO and VinePPO using four seeds per method. Every run uses one node of 8 H100 GPUs, so we compare methods directly in H100 hours. AC2 reaches the highest mean validation accuracy over seeds, 43.3% against 41.7% for GRPO, 40.0% for PPO and 40.5% for VinePPO. GRPO and PPO decline after their peaks while AC2 keeps improving. The results are in Figure 5 of the new Appendix A (Robustness across model, task, and seed) in the updated PDF.

We have made various other changes to the submitted PDF. All changes are in red so they can be found easily by reviewers. We provide specific responses to each reviewer below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.