acceptodds
Under review as a conference paper at ICLR 2027

Variational Self-Correction via Latent Diagnosis

Abstract

This paper studies post-hoc self-correction, in which an LLM revisits an initial response, identifies flaws in its reasoning, and revises the response before producing a final answer. Effective self-correction requires not only detecting what is wrong, but also translating that diagnosis into a successful revision without disrupting reasoning that is already correct. Existing methods, however, typically leave diagnosis implicit or optimize diagnosis and refinement in isolation, weakening the link between identified errors and the revisions they are intended to guide. To address this gap, we propose VaCrit, a variational self-correction framework that treats structured critique as a latent diagnosis connecting an initial response to its refinement. We first learn a critique prior through error-localization supervision and meta-utility learning, where sampled critiques are evaluated by the correction gains they induce. Building on this, we jointly optimize a critique posterior and a unified model for diagnosis and refinement under a shared variational objective, allowing refinement outcomes to assign credit to sampled critiques while preserving oracle-free diagnostic capability. Experiments on 7 challenging mathematical reasoning benchmarks show that VaCrit consistently outperforms existing methods across different model scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Correction, Variational Reasoning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CYj3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VaCrit, a framework for post-hoc self-correction in mathematical reasoning. It represents a textual critique as a latent diagnosis connecting an initial response to a corrected response. Training first develops a reference-free critic through error-localization supervision and refinement-utility learning. It then jointly optimizes a reference-conditioned critique posterior and a refinement model, while using the frozen critic as a prior and a distillation teacher to preserve diagnostic capability. At inference, only the final model is retained; the same checkpoint generates the initial response, diagnoses it, and produces a revision. Experiments with Qwen3-4B and Qwen3-8B on seven mathematical benchmarks report improvements in final accuracy and a favorable balance between error repair and preservation of initially correct answers.

Critical strengths
  1. The problem is relevant, and the evaluation considers both benefits and harms of revision. Improving self-correction requires repairing incorrect responses while avoiding damage to correct ones. Reporting correction gains, error-repair rates, and correctness-preservation rates provides a more informative evaluation than final accuracy alone.

  2. The complete system produces encouraging empirical improvements. Table 1 reports macro-average final accuracies of 49.10% and 50.48% for the 4B and 8B models, exceeding the strongest same-backbone baseline, SCoRe, by 2.33 and 1.91 percentage points. A single correction round improves accuracy on all seven benchmarks for both model sizes. The reported average preservation rates of initially correct responses are 97.28% and 96.64%, respectively.

  3. The paper provides useful implementation and feedback analyses. The algorithm, prompts, feedback extraction, and training settings make the intended procedure reasonably concrete. Table 3 also provides an informative finding: retaining the repair suggestion without the explicit error index preserves almost the full gain, whereas retaining the index alone provides essentially no net improvement. This helps identify which feedback content the trained refiner actually uses.

Critical weaknesses
  1. The conceptual contribution appears limited, and the incremental methodological advance is not sufficiently established. The paper's own related-work discussion describes feedback-guided correction and critics trained using downstream refinement outcomes. VaCrit combines these ideas with reference-conditioned feedback generation, variational regularization, and diagnostic distillation. The resulting training procedure is concrete, but the manuscript does not clearly establish what additional capability requires this particular formulation. Representing an intermediate critique as a latent variable and deriving a variational objective does not, by itself, demonstrate a substantial advance over existing critique-refinement training. The central question is whether the proposed objective provides a meaningful benefit over simpler ways of training the same diagnosis-and-correction pipeline; the current evidence does not resolve this question.

  2. The ablations do not isolate the components that constitute the main claimed contribution. Table 3 studies critic initialization and inference-time feedback fields. It does not isolate the reference-conditioned posterior, posterior-prior regularization, diagnostic self-distillation, or the advantage of jointly updating the posterior and refiner. Thus, the results support the effectiveness of the complete pipeline, but do not establish whether its gains require the proposed variational machinery or could be obtained with a simpler reference-assisted SFT or critique-refinement RL procedure. The error-index ablation further suggests that the explicit localization field adds little beyond the suggestion, although this does not imply that localization training is unnecessary.

  3. The connection between joint training and the diagnostic behavior actually deployed at inference remains insufficiently demonstrated. During joint training, correction outcomes directly optimize the reference-conditioned posterior qϕq_\phi, whereas the deployed model pθp_\theta learns its reference-free diagnostic behavior through distillation from the frozen prior pψp_\psi. The deployed diagnosis policy is not directly rewarded according to the usefulness of its own sampled critiques. Parameter sharing may still improve its diagnostic behavior, but this needs evidence. In particular, the ProcessBench results in Table 5 evaluate the prior before and after meta-utility training, rather than the final unified model after joint optimization. They therefore do not directly establish diagnostic improvement or retention through the joint-training stage.

  4. The relationship between the stated ELBO and the implemented objective needs clarification. Equation (5) contains KL(qϕ∥pθ)\mathrm{KL}(q_\phi\Vert p_\theta) for the deployed diagnosis distribution, while Equation (9) replaces this with KL(qϕ∥pψ)\mathrm{KL}(q_\phi\Vert p_\psi) and a separate distillation penalty KL(pψ∥pθ)\mathrm{KL}(p_\psi\Vert p_\theta). These terms are not interchangeable. In the stated special case β1=1,β2=0\beta_1=1,\beta_2=0, Equation (9) is naturally an ELBO for the mixture using the frozen prior, log⁡∑zpψ(z)sθ(z)\log\sum_z p_\psi(z)s_\theta(z), rather than the deployed mixture, log⁡∑zpθ(z)sθ(z)\log\sum_z p_\theta(z)s_\theta(z). The remark that Equation (9) “recovers the standard ELBO” should specify which marginal is bounded and explain its connection to the objective in Equation (1). The additional distillation penalty alone does not establish that the resulting objective bounds the deployed model's success probability.

  5. The computational cost of the improvements is not adequately characterized. Appendix D.2 specifies data alignment and decoding limits, but the paper does not provide an accuracy-versus-generated-tokens or comparable total-budget evaluation. VaCrit adds a diagnostic generation with thinking enabled, while initial generation and refinement use thinking-disabled modes. Training also involves multiple sampled critiques and refinements. Without reporting actual training and inference costs, it is difficult to assess how much benefit the proposed training scheme provides relative to simpler methods with a comparable computation budget.

Questions for authors
  1. Incremental contribution: What specific benefit does the proposed variational objective provide over a critique-refinement pipeline trained with filtered supervised trajectories or direct outcome-based RL? Can this benefit be demonstrated while controlling for backbone, training problems, access to reference solutions and localization labels, and computation budget?

  2. Core component ablations: What happens when reference conditioning is removed from the posterior, posterior-prior regularization is removed, or diagnostic distillation is removed? A direct critique-refinement RL baseline would also help establish the need for the variational formulation.

  3. Diagnostic capability at deployment: Can the authors evaluate the final reference-free model on ProcessBench and on its own generated responses, and compare it with the frozen critic? How large is the correction-performance gap when the refiner receives critiques from the posterior, the frozen prior, or the final model under their respective available inputs?

  4. Meaning of the variational bound: After replacing the diagnosis prior in Equation (5) by the frozen critic in Equation (9), what probability does the claimed ELBO bound? What assumptions or additional derivation connect it to the deployed model's correction probability?

  5. Efficiency: What are the measured training costs and average generated tokens per problem, broken down by generation, diagnosis, and refinement? Do the gains persist under a matched total inference budget?

7 comments
AuthorsOpenReview ↗

Response to Reviewer CYj3 - Part 1

[W1/Q1] The conceptual contribution appears limited, and the incremental methodological advance is not sufficiently established

The resulting training procedure is concrete, but the manuscript does not clearly establish what additional capability requires this particular formulation

Thank you for asking us to clarify the methodological benefit. VaCrit is designed to learn reference-informed critiques that adapt to the current refiner, while retaining reference-free diagnosis in the deployed model. Our reference-conditioned posterior learns which critiques induce successful refinements as the refiner changes, while the learned reference-free critic regularizes the posterior and anchors the unified model’s diagnostic behavior. The intended benefit is more effective error repair while preserving already-correct responses.

whether the proposed objective provides a meaningful benefit over simpler ways of training the same diagnosis-and-correction pipeline

We evaluate this benefit through an existing critique-guided RL baseline and a controlled objective comparison. (i) CURE, included in our main experiments, provides a comparison with an established critique-guided RL approach that jointly trains critique generation and guided solving using downstream outcomes. (ii) Joint correctness-RL replaces the expected log-success objective Eqϕ[log⁡sθ]\mathbb{E}{q_\phi}[\log s_\theta] with expected correctness Eqϕ[sθ]\mathbb{E}{q_\phi}[s_\theta] in both posterior and unified-model updates, where sθs_\theta denotes the probability of a correct refinement given a critique. This control retains reference conditioning, joint updates, and both KL terms. In this way, the control implements joint outcome-based RL under the same supervision and regularization, allowing us to assess whether the variational log-success objective provides an additional benefit. VaCrit and Joint correctness-RL share the same training problems, initialization, access to reference solutions and localization supervision, optimization settings, and rollout counts (K=M=3)(K=M=3), with β1=β2=0.1\beta_1=\beta_2=0.1.

All three methods use Qwen3-4B and the same training data. Results are unweighted averages across seven benchmarks, with each model generating its own initial responses and performing correction without reference solutions.

MethodInitialFinalGainCSCL
VaCrit44.8049.10+4.3013.1997.28
Joint correctness-RL44.0647.67+3.6110.7497.37
CURE45.7745.71−0.0616.8282.11

VaCrit’s stronger correctness preservation than CURE allows successful repairs to translate into a positive net correction gain. This is consistent with diagnostic distillation anchoring the unified model’s critique behavior to the learned critic during joint training, which can help limit harmful revisions of correct answers as the refiner adapts.

Against Joint correctness-RL, VaCrit improves error repair at comparable correctness preservation. For a given critique, log-success assigns greater value to the same increase in correction probability when that probability is low, encouraging improvement on less successful cases. This provides a rationale consistent with the stronger error repair observed here. The complementary roles of reference conditioning, posterior adaptation, joint updates, and diagnostic distillation are examined in W2/Q2.

AuthorsOpenReview ↗

Response to Reviewer CYj3 - Part 2

[W2/Q2] The ablations do not isolate the components that constitute the main claimed contribution

Thank you for requesting more targeted ablations. We address the requested critique-refinement RL comparison in W1/Q1 through CURE and Joint correctness-RL, with the latter directly optimizing expected correctness under the same supervision and regularization. Here, we complement these comparisons with four training ablations: (i) w/o reference conditioning removes the reference solution from the posterior’s input while retaining posterior learning; (ii) Frozen posterior freezes the posterior at the common initialization while continuing to train the unified model; (iii) Sequential training first trains the posterior with the unified model fixed, then trains the unified model with the learned posterior fixed; and (iv) w/o diagnostic distillation sets β2=0\beta_2=0 while retaining the remaining objectives.

All variants in the table use Qwen3-4B, the same critic initialization, and the same training data. Results are unweighted averages across seven benchmarks, with each final model generating its own initial responses, reference-free critiques, and refinements.

MethodInitialFinalGainCSCL
VaCrit (full)44.8049.10+4.3013.1997.28
w/o reference conditioning44.0847.71+3.6312.6696.90
Frozen posterior44.7648.26+3.5011.4297.05
Sequential training44.4947.65+3.1611.0696.34
w/o diagnostic distillation46.7050.47+3.7713.9395.80

Reference conditioning and joint adaptation contribute beyond critic initialization. Removing reference conditioning weakens correction, consistent with references providing additional evidence for identifying errors and repair directions. Freezing the posterior reduces correction gain despite nearly identical initial accuracy, and sequential training also underperforms joint updates. Joint updates allow critique generation and refinement to adapt to one another, which can help keep feedback useful as the refiner changes.

Diagnostic distillation improves correctness preservation and net correction gain. Removing it yields higher final accuracy alongside stronger initial responses, but a smaller improvement through correction. Retaining distillation anchors the unified model’s diagnostic behavior to the learned critic during joint training, which can help limit harmful revisions of initially correct answers.

AuthorsOpenReview ↗

Response to Reviewer CYj3 - Part 3

[W3/Q3] The connection between joint training and the diagnostic behavior actually deployed at inference remains insufficiently demonstrated

Can the authors evaluate the final reference-free model on ProcessBench and on its own generated responses, and compare it with the frozen critic?

Thank you for highlighting the need for direct evidence on the deployed model. We evaluate the final unified model after joint training, examining diagnostic retention on ProcessBench and the correction utility of its own critiques. All evaluations below use Qwen3-4B.

First, we assess diagnostic retention on ProcessBench by comparing the final model pθp_\theta with the frozen prior pψp_\psi obtained after meta-utility training. Both receive no reference solutions and follow identical evaluation settings within each output format. Entries report F1 scores; Avg. is the unweighted mean across the four subsets.

FormatModelGSM8KMATHOlympiadOmniMATHAvg.
XMLFrozen prior pψp_\psi84.8984.2379.5874.4580.79
XMLFinal model pθp_\theta84.0584.7580.2073.5280.63
MarkdownFrozen prior pψp_\psi85.9982.7478.1273.1480.00
MarkdownFinal model pθp_\theta85.6983.9576.8573.4779.99

As shown in the table above, the final model largely retains the frozen prior’s average diagnostic performance in both formats. This is consistent with initializing the unified model from the learned critic and maintaining diagnostic distillation during joint training, which penalizes departures from the critic’s diagnostic behavior as the shared model learns refinement.

How large is the correction-performance gap when the refiner receives critiques from the posterior, the frozen prior, or the final model under their respective available inputs?

To complement this diagnostic evaluation, we test whether the final model’s reference-free critiques provide useful feedback for correcting its own generated responses. We fix the initial responses generated by the final model, the final refiner, decoding settings, and generation budgets, changing only the critique source. The frozen prior and final model receive identical reference-free inputs. The posterior qϕq_\phi additionally receives the reference solution y⋆y^\star, which is unavailable at deployment.

Initial accuracy is fixed at 86.40% on MATH-500 and 54.15% on OlympiadBench. Entries report final accuracy, with correction gain in percentage points in parentheses; bold identifies the better reference-free result.

Critique sourceReference solutionMATH-500OlympiadBench
Frozen prior pψp_\psiNo89.00 (+2.60)58.46 (+4.31)
Final model pθp_\thetaNo90.00 (+3.60)59.05 (+4.90)
Posterior qϕq_\phiYes90.60 (+4.20)68.25 (+14.10)

As shown in the table above, the final model’s own critiques outperform frozen-prior feedback under identical reference-free conditions. With initial responses and the refiner held fixed, this comparison directly assesses the usefulness of the supplied feedback to the deployed refiner. The frozen prior was optimized with a frozen base-model refiner, whereas the unified model’s shared parameters continue to adapt during joint training, potentially making its own critiques better suited to the final refiner. With access to reference solutions unavailable at deployment, posterior feedback exceeds self-feedback by 0.60 and 9.20 percentage points on MATH-500 and OlympiadBench, respectively.

Reviewer_QjzVOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VaCrit, a self-correction method for LLM math reasoning in which a structured critique (an error index plus a natural-language repair suggestion) is treated as an explicit latent variable mediating the initial and refined responses. An oracle-free critic prior is trained by RL, first for error localization against PRM800K step labels and then for critique utility measured as the correctness gain of a critique-guided refinement over an unguided one. A reference-conditioned posterior and a unified critic-plus-refiner are then trained jointly under an ELBO-style objective with two KL terms, one of which distills the frozen critic's critique distribution into the unified model so that diagnosis survives without the reference at inference. Evaluation on Qwen3-4B and 8B covers MATH-500, AMC23, AIME24 to AIME26, Minerva, and OlympiadBench, reporting mean self-correction gains of 4.30 and 4.12 points against 1.16 and 1.08 for SCoRe, together with multi-round curves and RSS, CS, and CL diagnostics. ProcessBench is used for localization F1.

Critical strengths

The empirical gains over SCoRe are large and consistently signed across seven benchmarks and two scales, and the paper is unusually transparent about its own evaluation asymmetries. The inference-side ablation design is clean: the ablated variants reuse the same first response and the same critique, so the initial accuracy is held fixed at 44.80 across all inference ablations, which makes the component attributions trustworthy rather than confounded by different starting distributions. The multi-round experiment is also reported honestly, including that SCoRe's curve is nearly flat under the restricted protocol.

Critical weaknesses

The paper's framing rests on explicit diagnosis, but the ablation shows the diagnosis is inert. Removing the error index while keeping the repair suggestion changes final accuracy from 49.09 to 49.10, and the ProcessBench numbers move by less than one F1 point across the meta-utility stage. The authors note this in passing, but it means the localization stage, the PRM800K data, and the structured critique interface are not what produces the result, and the abstract and Figure 1 do not reflect this. Second, the methods closest to the paper's claimed axis are cited at length and never run. CTRL, RefCritic, Critique-RL, RCO, and DeepCritic all couple diagnosis to refinement through downstream outcomes or judgment losses, which is exactly what the joint objective here does; the baselines actually evaluated, STaR+, SCoRe, PAG, and CURE, are direct or self-verification methods. Without at least one coupled-diagnosis baseline, the improvement cannot be attributed to the coupling. There are related smaller issues: with beta1 set to 0.1 the optimized objective is not a lower bound on the stated objective, so the variational framing does not apply to the configuration actually run; beta2 is never ablated; no seeds, standard deviations, or significance tests are reported anywhere; the seven-benchmark average mixes 32 samples per problem on AMC23 and the AIME sets with a single sample on MATH-500, Minerva, and OlympiadBench; and the reported checkpoint is step 200 of a maximum 300 without the step-300 number or an explanation.

Questions for authors

With beta1 at 0.1 the objective is not an ELBO. What happens at beta1 equal to 1, and is the variational treatment load-bearing, or is this equivalent to sampling K critiques and applying REINFORCE to the one that produced a correct refinement? What is the result with beta2 set to zero, given that self-distillation is proposed as the mechanism preserving oracle-free diagnosis? Given that the error index is worth 0.01 accuracy points, is it needed at inference at all, and what justifies the localization stage? On Minerva the method is not the best at either scale, and its first-response accuracy there is the lowest of all methods, so part of the headline gain appears to sit on a depressed floor. Figure 2 restricts SCoRe and PAG to a restricted context; would the authors rerun with the original formulations? Please report multiple seeds, and report the fraction sweep used for hyperparameter selection.

8 comments
AuthorsOpenReview ↗

Response to Reviewer QjzV - Part 1

[W1] Contributions of localization training and diagnostic feedback

The paper's framing rests on explicit diagnosis, but the ablation shows the diagnosis is inert

Thank you for your feedback. We address this concern through a training-stage ablation and an expanded ProcessBench evaluation, then clarify what the error-index ablation measures.

Contribution of localization training. We compare the full pipeline against a variant that removes the PRM800K-based localization warm-up while retaining meta-utility and joint training. These two conditions use identical downstream training data, schedules, and hyperparameters. All variants use Qwen3-4B; the table reports unweighted averages across the seven correction benchmarks.

MethodInitialFinalGainCSCL
VaCrit (full)44.8049.10+4.3013.1997.28
w/o localization43.9446.90+2.9512.3094.78
w/o meta-utility44.0447.64+3.6012.9796.49
w/o critic training44.0247.16+3.1410.9596.58

As shown in the table above, removing localization lowers final accuracy on all seven benchmarks, with an average reduction of approximately 2.2 percentage points. Localization warm-up explicitly trains the critic to identify the first erroneous step and recognize error-free responses, providing a diagnostic foundation for subsequent meta-utility and joint training. This initialization can help feedback target genuine errors while avoiding unnecessary revisions to correct responses, consistent with the higher error-repair success and correctness preservation observed in the full model.

Diagnostic performance across training stages. To complement these downstream results, we evaluate three checkpoints on ProcessBench: the base model, after localization, and after meta-utility training. This comparison measures diagnostic improvements from localization and their retention after training for correction utility. Within each output format, inputs, prompts, parsing, and decoding settings are identical across checkpoints. Entries report F1 scores; Avg. is the unweighted mean across the four subsets.

FormatModelGSM8KMATHOlympiadOmniMATHAvg.
XMLBase model84.7583.1577.4872.0879.37
XMLAfter localization85.4984.1577.8474.0180.37
XMLAfter meta-utility84.8984.2379.5874.4580.79
MarkdownBase model83.4881.3775.9171.5278.07
MarkdownAfter localization83.9782.2178.2772.7379.30
MarkdownAfter meta-utility85.9982.7478.1273.1480.00

Localization improves diagnostic F1 on all four subsets in both formats, including Markdown despite XML-only localization supervision. Both checkpoints in the original before/after-meta comparison had already completed localization, so that comparison does not measure the contribution of this earlier stage.

Moreover, ProcessBench measures first-error localization and error-free recognition, whereas meta-utility training optimizes critiques for downstream correction. Its modest F1 improvement therefore does not capture its full contribution to refinement: removing meta-utility training lowers average final accuracy from 49.10% to 47.64% in the first table.

Removing the error index while keeping the repair suggestion changes final accuracy from 49.09 to 49.10, and the ProcessBench numbers move by less than one F1 point across the meta-utility stage

Role of the explicit error index. The repair suggestion can identify the erroneous statement or operation and explain how to correct it, preserving diagnostic information when the separate numerical index is removed. For example, the suggestion in Appendix E identifies the integer omitted from the student’s list and requests a recount, indicating what needs revision without relying on the separate step number.

AuthorsOpenReview ↗

Response to Reviewer QjzV - Part 2

[W2] Comparisons with coupled-diagnosis methods and training controls

Thank you for highlighting the importance of comparisons with coupled-diagnosis methods. We respectfully point out that CURE, included in our experiments, couples critique learning to downstream correction outcomes. It generates error diagnoses and hints, rewards critiques using subsequent regeneration accuracy, and jointly optimizes solving, critiquing, and guided regeneration within one policy. In Table 1, VaCrit achieves higher average final accuracy than CURE by 3.39 and 5.24 percentage points with Qwen3-4B and Qwen3-8B, respectively.

To complement this comparison with an existing coupled-diagnosis method, we examine the contributions of VaCrit’s training objective, continued posterior learning, and joint updates using three internal controls: (i) Joint correctness-RL: Replace Ez∼qϕ(⋅∣x,y1,y⋆)[log⁡sθ(x,y1,z)]\mathbb{E}_{z\sim q_\phi(\cdot\mid x,y_1,y^\star)}[\log s_\theta(x,y_1,z)] with Ez∼qϕ(⋅∣x,y1,y⋆)[sθ(x,y1,z)]\mathbb{E}_{z\sim q_\phi(\cdot\mid x,y_1,y^\star)}[s_\theta(x,y_1,z)] in both posterior and unified-model updates, retaining reference conditioning, joint updates, and both KL terms. (ii) Frozen posterior: Freeze qϕ(z∣x,y1,y⋆)q_\phi(z\mid x,y_1,y^\star) at the common initialization and continue training pθp_\theta. (iii) Sequential training: First train qϕq_\phi with pθp_\theta fixed, then freeze the learned posterior and train pθp_\theta. All variants use Qwen3-4B, the same critic initialization, and the same training data. All metrics are macro-averaged across the seven benchmarks, with equal weight assigned to each benchmark.

MethodInitialFinalGainCSCL
VaCrit (full)44.8049.10+4.3013.1997.28
Joint correctness-RL44.0647.67+3.6110.7497.37
Frozen posterior44.7648.26+3.5011.4297.05
Sequential training44.4947.65+3.1611.0696.34

As shown in the table above, VaCrit achieves higher final accuracy than all three controls. The log-success objective penalizes critiques with low refinement success probability more strongly, while giving greater weight to improving the refiner’s performance under such feedback. This encourages the posterior to avoid ineffective critiques and the refiner to learn from cases it handles less successfully, consistent with the higher error-repair success observed relative to Joint correctness-RL at comparable correctness preservation.

Joint training allows the posterior to adapt its critiques to the refiner’s changing capabilities. A frozen posterior cannot make this adjustment, which can limit correction despite nearly unchanged initial accuracy. Sequential training updates both components, but learns the posterior against a fixed refiner and then freezes it during refinement learning. Joint updates maintain this interaction throughout training, providing a rationale for the higher error-repair success and correctness preservation observed over sequential training.

AuthorsOpenReview ↗

Response to Reviewer QjzV - Part 3

[W3] Variational justification at β1=0.1\beta_1=0.1

Thank you for raising this point. At β1=0.1\beta_1=0.1, the objective in Eq. (9) remains proportional to a valid variational lower bound under the frozen critique prior pψp_\psi, and its connection to the original objective can be made explicit.

Specifically, let sθ(x,y1,z)=pθ(Ax∣x,y1,c(z))∈[0,1]s_\theta(x,y_1,z)=p_\theta(A_x\mid x,y_1,c(z))\in[0,1], and write ED\mathbb E_D for expectation over (x,y1,y⋆)∼D(x,y_1,y^\star)\sim D. For 0<β1≤10<\beta_1\leq1 and β2≥0\beta_2\geq0, the exact objective satisfies:

\leq \mathbb E_D \left[ \log\mathbb E_{z\sim p_\psi(\cdot\mid x,y_1)} [s_\theta(x,y_1,z)] \right].$$ This follows from $\log s_\theta\leq0$, the nonnegativity of the diagnostic-distillation KL, and the ELBO inequality with prior $p_\psi$. Thus, at $\beta_1=0.1$, **maximizing $J_{\mathrm{uni}}$ is equivalent to maximizing the frozen-prior lower bound $10J_{\mathrm{uni}}$**. To connect this bound to the original objective, we compare prior critiques and the model’s own critiques under the same inputs and refiner. Applying KL data processing to the binary correctness outcome and bounding the possible decrease in log success probability gives: $$\begin{aligned} \mathbb E_D[\log \mathbb{E}_{z \sim p_\theta(\cdot \mid x, y_1)}[s_\theta(x, y_1, z)]] \geq{}& \frac{J_{\mathrm{uni}}(\theta,\phi)}{\beta_1} -\mathbb E_D\Bigg[ \frac{ D_{\mathrm{KL}}\left( p_\psi(\cdot\mid x,y_1) \Vert p_\theta(\cdot\mid x,y_1) \right) }{ \mathbb E_{z\sim p_\psi(\cdot\mid x,y_1)} [s_\theta(x,y_1,z)] } + \sqrt{ \frac{ 2D_{\mathrm{KL}}\left( p_\psi(\cdot\mid x,y_1) \Vert p_\theta(\cdot\mid x,y_1) \right) }{ \mathbb E_{z\sim p_\psi(\cdot\mid x,y_1)} [s_\theta(x,y_1,z)] } } \Bigg]. \end{aligned}$$ This holds when the correction probability under $p_\psi$ is positive for each input, the KL over complete critique sequences is finite, and the displayed expectations are finite. **The transfer term depends on the diagnostic KL already penalized in Eq. (9) and the correction probability under $p_\psi$, and vanishes when the conditional prior and self-critique distributions coincide.** The same objective also clarifies how $\beta_1$ balances the learned prior against correction utility. For a fixed input and refiner, maximizing the weighted objective over all critique distributions yields: $$q_{\beta_1}^\star(z\mid x,y_1,y^\star) \propto p_\psi(z\mid x,y_1) s_\theta(x,y_1,z)^{1/\beta_1}.$$ Thus, $\beta_1=0.1$ weights critiques by $s_\theta^{10}$, compared with $s_\theta$ at $\beta_1=1$, placing greater emphasis on critiques that enable successful refinement while retaining the learned prior as a reference. To evaluate this choice under deployment, we compare $\beta_1=0.1$ and $\beta_1=1$ using Qwen3-4B, holding $\beta_2=0.1$, initialization, and all remaining training and evaluation settings fixed. Each final model generates its own initial responses, critiques, and refinements. Results are unweighted averages across seven benchmarks. | $\beta_1$ | Initial | Final | Gain | CS | CL | | --- | --- | --- | --- | --- | --- | | 0.1 | 44.80 | **49.10** | **+4.30** | **13.19** | 97.28 | | 1 | 45.07 | 49.07 | +4.00 | 12.36 | 97.33 | As shown in the table above, **the default $\beta_1=0.1$ achieves stronger error repair while maintaining comparable correctness preservation**, consistent with giving the posterior greater flexibility to favor critiques useful to the current refiner. **Increasing $\beta_1$ to 1 retains comparable final accuracy and positive correction gains on all seven benchmarks**, supporting robustness to this tenfold change in regularization strength.
Reviewer_GUwfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper proposes VaCrit, a self-correction method that uses structured critiques to connect an initial answer with its revision. Each critique identifies the first incorrect step and provides a repair suggestion.
  • The method first trains a critic through error localization and rewards based on revision outcomes. It then jointly trains a reference-conditioned critique posterior and a shared diagnosis/refinement model, using the frozen critic to preserve diagnostic ability. Reference solutions are only available during training.
  • Experiments cover seven math benchmarks with Qwen3-4B and Qwen3-8B. VaCrit reaches average final accuracies of 49.10% and 50.48%, beating the strongest same-backbone baseline by 2.33 and 1.91 percentage points. Single-round revision improves its own initial accuracy by 4.30 and 4.12 points. The paper also reports further gains over four refinement rounds on four benchmarks.
Critical strengths
  • The method targets a useful problem that is well-motivated: making critiques lead to successful repairs. Accurate error detection alone does not guarantee a better answer. Training feedback based on its effect on revision directly addresses this gap. Using one checkpoint for generation, diagnosis, and refinement also avoids needing a separate critic at deployment.
  • The results show a good balance between repairing mistakes and preserving correct answers. Revision improves accuracy on all seven benchmarks for both model sizes. Reporting both repair and preservation makes the evidence more useful than final accuracy alone.
Critical weaknesses
  • I am not yet convinced which part of the new training setup earns the gains. SCoRe already trains self-correction through RL, and CTRL already trains critiques through their effect on a refiner. The distinctive parts here are the reference-conditioned critique model, joint training, and diagnostic distillation. Table 3 does not remove those components. Its feedback ablations show that repair suggestions matter, but that is different from showing why the proposed training method is better.
  • The gap between training-time and test-time feedback needs more evidence. The refiner learns from critiques informed by reference solutions, but has to rely on its own critiques at test time. Distillation is meant to bridge this gap. However, Table 5 evaluates the critic before and after meta-utility training, not the final unified model after joint training. I would like direct evidence that the final model retains the diagnostic ability the method relies on.
Questions for authors
  • How much performance remains when the same pipeline is trained using reference-free critic samples instead of the reference-conditioned critique model? What happens when diagnostic distillation is removed, with the training rollout budget held fixed?
  • What is the final unified model's ProcessBench performance after joint training? Please also clarify why the objective built around the frozen critic is the right justification for the final model's own test-time critiques.
  • Does the multi-round advantage remain when SCoRe and PAG retain their original interaction history, rather than the restricted history used in Section 4.5? What does the comparison look like at a matched generated-token budget?
4 comments
AuthorsOpenReview ↗

Response to Reviewer GUwf - Part 1

[W1/Q1] Contributions of the core training components

Thank you for highlighting the need to isolate the sources of the gains. To address this concern, we added four training ablations to directly assess the contributions of the components identified in your comment: (i) w/o reference conditioning removes the reference solution from the posterior’s input while retaining posterior learning and the remaining objectives; (ii) Frozen posterior freezes the posterior at the common initialization while continuing to train the unified model; (iii) Sequential training first trains the posterior with the unified model fixed, then trains the unified model with the learned posterior fixed; and (iv) w/o diagnostic distillation removes only the diagnostic-distillation term by setting β2=0\beta_2=0.

For these comparisons, all variants use Qwen3-4B, the same critic initialization, and the same training data. All metrics are macro-averaged across the seven benchmarks, with equal weight assigned to each benchmark.

MethodInitialFinalGainCSCL
VaCrit (full)44.8049.10+4.3013.1997.28
w/o reference conditioning44.0847.71+3.6312.6696.90
Frozen posterior44.7648.26+3.5011.4297.05
Sequential training44.4947.65+3.1611.0696.34
w/o diagnostic distillation46.7050.47+3.7713.9395.80

As shown in the table above, removing reference conditioning reduces final accuracy and correction gain. Freezing the posterior reduces correction gain despite nearly unchanged initial accuracy, supporting continued posterior adaptation. Sequential training also lowers both CS and CL, supporting joint rather than separate-stage updates. Removing diagnostic distillation yields higher final accuracy alongside a stronger starting point, but reduces correctness preservation and net correction gain.

Taken together, these results support the proposed training design: reference solutions guide critique learning, while joint updates continually adapt the critiques to the refiner learning to use them. Diagnostic distillation complements this adaptation by helping preserve initially correct answers, yielding the largest net correction gain among the evaluated training variants.

AuthorsOpenReview ↗

Response to Reviewer GUwf - Part 2

[W2] Diagnostic retention and self-feedback utility after joint training

Thank you for pointing out this evaluation gap. We directly evaluate the final unified model after joint training through two complementary assessments: diagnostic retention and the correction utility of its own critiques.

First, we evaluate diagnostic retention on ProcessBench, comparing the frozen prior pψp_\psi, obtained after meta-utility training, with the final VaCrit model pθp_\theta. Both use Qwen3-4B, receive no reference solutions, and follow the same evaluation protocol within each output format. Entries report F1 scores.

FormatModelGSM8KMATHOlympiadOmniMATHAvg.
XMLFrozen prior84.8984.2379.5874.4580.79
XMLVaCrit84.0584.7580.2073.5280.63
MarkdownFrozen prior85.9982.7478.1273.1480.00
MarkdownVaCrit85.6983.9576.8573.4779.99

As shown in the table above, the final model’s average F1 remains close to the frozen prior in both formats, demonstrating that diagnostic performance is largely retained through joint training.

Second, we complement this diagnostic assessment by testing whether the final model’s own critiques help correct its generated responses. We fix the initial responses generated by the final VaCrit model, the final refiner, decoding settings, and generation budgets, changing only the critique source. The frozen prior and final model both operate without reference solutions. We additionally include the posterior qϕq_\phi as a reference-assisted comparison: it receives the reference solution y⋆y^\star, which is unavailable at deployment.

Initial accuracy is fixed at 86.40% on MATH-500 and 54.15% on OlympiadBench. Entries report final accuracy, with correction gain in percentage points in parentheses. Bold indicates the better result among the reference-free sources.

Critique sourceFeedback settingMATH500Olympiad
Frozen prior pψp_\psiReference-free89.00 (+2.60)58.46 (+4.31)
VaCrit pθp_\thetaReference-free90.00 (+3.60)59.05 (+4.90)
Posterior qϕq_\phiReference-assisted90.60 (+4.20)68.25 (+14.10)

With the initial responses and refiner held fixed, the final model’s own critiques outperform frozen-prior feedback on both benchmarks without access to reference solutions. The frozen prior was optimized for correction utility with a frozen base model refiner, whereas the unified model’s shared parameters continue to adapt during joint training, potentially making its own critiques better suited to the final refiner. Together with the ProcessBench results, this shows that the final unified model retains diagnostic performance and provides useful feedback on its own generated responses after joint training.

AuthorsOpenReview ↗

Response to Reviewer GUwf - Part 3

[Q2] Connecting the frozen-critic objective to test-time self-feedback

What is the final unified model's ProcessBench performance after joint training?

Thank you for asking us to clarify this connection. The final unified model achieves average ProcessBench F1 scores of 80.63% in XML and 79.99% in Markdown, largely retaining the frozen prior’s diagnostic performance. The complete comparison appears in the first table of our response to W2.

Please also clarify why the objective built around the frozen critic is the right justification for the final model's own test-time critiques.

We equip the critic prior with reference-free diagnostic capability through error-localization training and optimize its feedback utility through meta-utility learning with a frozen base-model refiner. Since the final model retains comparable diagnostic performance and frozen-prior feedback remains useful on the final model’s own responses (correction gains of 2.60 and 4.31 percentage points on MATH-500 and OlympiadBench, respectively; second table in our response to Weakness 2), this learned critic remains a relevant training reference for the model’s test-time critiques.

To further justify this role in our objective, we connect posterior-feedback training to correction under the model’s own feedback. For fixed (x,y1,y⋆)(x,y_1,y^\star) and the same current refiner, let sθ(x,y1,z)=pθ(Ax∣x,y1,c(z))∈[0,1]s_\theta(x,y_1,z)=p_\theta(A_x\mid x,y_1,c(z))\in[0,1] denote the probability of a correct refinement. By Jensen’s inequality, exponentiating the expected log-success gives a lower bound on the expected correction probability under posterior feedback. We then bound how much this probability can change when posterior feedback is replaced by self-feedback. For complete critique-sequence distributions, the total variation triangle inequality and Pinsker’s inequality give:

\Big| \mathbb{E}_{z \sim q_\phi(\cdot \mid x, y_1, y^\star)}[s_\theta(x, y_1, z)] - \mathbb{E}_{z \sim p_\theta(\cdot \mid x, y_1)}[s_\theta(x, y_1, z)] \Big| \le \sqrt{\frac{1}{2}D_\text{KL}(q_\phi(\cdot \mid x, y_1, y^\star) \parallel p_\psi(\cdot \mid x, y_1))} + \sqrt{\frac{1}{2}D_\text{KL}(p_\psi(\cdot \mid x, y_1) \parallel p_\theta(\cdot \mid x, y_1))} \end{aligned}$$ This bound makes the role of the frozen critic explicit: it serves as a common reference through which the two KL divergences control the change in correction success between posterior feedback and self-feedback. The log-success term in Eq. (9) encourages successful correction under posterior feedback, while the two regularizers penalize the discrepancies governing this feedback replacement. **When the corresponding KL divergences are small, this provides a conditional justification for the training design.** To evaluate the model’s own feedback in practice, the **second table in our response to Weakness 2** holds the final model’s initial responses, refiner, decoding settings, and generation budgets fixed, changing only the critique source. **The final model’s own reference-free critiques outperform frozen-prior feedback on both benchmarks**, complementing the conditional argument with direct evidence of useful self-feedback at deployment.
AuthorsOpenReview ↗

Response to Reviewer GUwf - Part 4

[Q3] Evaluation with original interaction histories and matched token caps

Thank you for raising these concerns. We compare SCoRe and PAG on AMC23 using Qwen3-4B under both the restricted-history setting in Section 4.5 and their original interaction-history protocols. Within each baseline, the checkpoint, initial responses, and decoding settings remain unchanged across history conditions, and its original stopping rule is preserved. VaCrit follows the protocol in Section 4.5. The table reports accuracy for the initial response and four successive correction rounds.

MethodInitial (y1y_1)Round 1 (y2y_2)Round 2 (y3y_3)Round 3 (y4y_4)Round 4 (y5y_5)
VaCrit69.6174.3876.0976.7277.34
SCoRe (restricted history)70.5571.6471.7271.8071.72
SCoRe (original history)70.5571.6472.0372.0372.11
PAG (restricted history)68.9869.6169.6169.4569.61
PAG (original history)68.9869.6168.7568.4468.44

As shown in the table above, VaCrit retains its multi-round advantage when SCoRe and PAG use their original interaction histories, continuing to improve across four correction rounds and finishing 5.23 percentage points ahead of the strongest baseline on AMC23.

To address the generation-budget concern, we additionally evaluate single-round correction on OlympiadBench using Qwen3-4B under total correction-token caps of 4k, 8k, and 16k. Each method reuses its own fixed initial responses across caps; initial-response generation is excluded from these correction budgets. SCoRe allocates the entire cap to refinement. PAG and VaCrit split it equally between verification/diagnosis, including all thinking tokens, and refinement, without transferring unused allowances between stages. The table reports final accuracy.

Total correction-token capSCoRePAGVaCrit
4k55.9353.2655.93
8k55.7952.9759.05
16k55.4953.1259.05

As shown in the table above, VaCrit at an 8k correction cap outperforms SCoRe at 16k, with both diagnostic thinking and refinement included in VaCrit’s allowance. Increasing the allowance for direct refinement therefore does not close the accuracy gap in this comparison.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised PDF uploaded

We have uploaded the revised PDF, with newly added content highlighted for ease of reference. The main updates are:

  • Theoretical clarification: clarified the relationship between the training objective and the frozen-prior ELBO, including the scaling required by our default KL coefficient; added a conditional transfer bound connecting frozen-prior and deployed critique utility (Sec. 3.3; App. C.4).
  • Additional baseline: added a comparison with RefCritic retrained on Qwen3-4B using LoRA, reporting mean performance across seven benchmarks (App. D.7, Table 12).
  • Training objective and ablations: added a controlled comparison with expected-correctness RL, ablations of reference conditioning, posterior learning, joint training, and diagnostic distillation, and KL-coefficient sensitivity analysis, including removal of the posterior-to-prior KL penalty (β1=0\beta_1=0) (Table 3; App. D.8, Table 13).
  • Diagnostic retention and critique utility: added final-model error-localization results on XML and Markdown, and compared frozen-prior, self-generated, and reference-conditioned posterior critiques with initial responses and the refiner held fixed (App. D.4, Table 6; App. D.9, Table 14).
  • Evaluation robustness and protocols: added five-seed evaluations with fixed checkpoints and an AMC23 comparison using SCoRe’s and PAG’s original history-retention protocols; clarified checkpoint selection, sampling, and metric aggregation (Apps. D.2, D.10–D.11; Tables 15–16).
  • Compute budgets and costs: added comparisons under matched correction-token caps, including thinking tokens, and reported actual generated-token counts and training GPU-hours (App. D.12; Tables 17–18).

We thank the reviewers for their constructive feedback and their time in reviewing these updates.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.