acceptodds
Under review as a conference paper at ICLR 2027

STAC: Evaluator-Free Medical Multimodal RLVR via Stability-Aware Credit Assignment

Abstract

Although reinforcement learning with verifiable rewards (RLVR) has shown promise in medical multimodal large language models, binary outcome supervision cannot distinguish robust reasoning from fragile successes that arrive at the correct answer by chance. While process reward models offer finer-grained guidance, they introduce heavy annotation or external evaluator overhead. In this work, we reveal that the rollout policy’s own intermediate states inherently reflect trajectory stability: early prefix support for the eventual answer reliably predicts continuation stability under resampling, serving as a lightweight process signal orthogonal to final correctness. Grounded in this insight, we propose STAC (STability-Aware Credit Assignment), an evaluator-free framework that refines credit assignment using internal policy dynamics. STAC introduces Prefix Support Probing to evaluate early intermediate states against competing hypotheses, and couples it with terminal verification via Outcome-Conditioned Reward Modulation to reward stably supported correct reasoning while suppressing credit for stubborn errors. Crucially, STAC requires no external judges, step-level annotations, or rollout branching during training. Across six diverse medical visual question answering benchmarks, STAC consistently outperforms strong RLVR baselines, significantly improving both diagnostic accuracy and reasoning stability.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Large Language Models, RLVR, medical

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VxG5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

STAC augments GRPO for medical multimodal RLVR by probing an early reasoning prefix and using the policy’s relative support for its eventual answer as a stability signal. This signal is conditioned on terminal correctness to reward stably correct rollouts and penalize stubbornly incorrect ones. Experiments on six medical VQA benchmarks with Qwen3-VL-4B report small gains over GRPO and CARE.

Critical strengths

The pilot study is carefully designed, and the negative result on entropy-guided localization is useful.

The “same answer, different stability” framing is clinically intuitive and makes the limitation of outcome-only RLVR concrete.

The ablation on Shuffled Support and Direct Prefix Support shows that both trajectory alignment and outcome conditioning matter.

The method adds only one parallel forward pass and avoids external PRMs or continuation branching, which is a practical advantage.

Critical weaknesses

W1. The gains are marginal and lack statistical support.
STAC improves average accuracy by only 0.9 points over GRPO (62.4 vs. 61.5), smaller than several per-dataset standard deviations, and no confidence intervals or significance tests are reported.

W2. The prefix-support signal is noisy and uncalibrated.
The pilot shows only weak within-query correlation with continuation stability (ρ≈0.36\rho \approx 0.36–0.420.42) and near-chance correctness concordance (C≈0.51C \approx 0.51–0.560.56), yet STAC treats the rescaled support as a reliable trajectory-level score without analyzing fallback frequency or ranking stability.

W3. The distinction from confidence-aware RLVR is not experimentally isolated.
CARE also modulates a policy-internal signal by terminal correctness; the paper does not test whether prefix support beats a simple confidence proxy (e.g., rollout average log-probability) under the same reward structure.

W4. Generalization is under-supported.
All main results use Qwen3-VL-4B; Zero-RL gains shrink from 0.6 points at 2B to 0.5 points at 4B, larger scales (7B–8B) are not evaluated, and the relation to MedVR’s consensus-based credit assignment is not discussed.

Questions for authors
  1. What fraction of training examples trigger the neutral fallback P~=0.5\tilde{P}=0.5? When it does not trigger, how large is Pmax⁡−Pmin⁡P_{\max}-P_{\min} typically?
  2. Can the authors provide a controlled variant where prefix support is replaced by a terminal-confidence proxy under the same outcome-conditioned modulation?
  3. Why were 7B–8B backbones not evaluated, given the shrinking gains from 2B to 4B?
5 comments
AuthorsOpenReview ↗

Response to Reviewer VxG5 [1/3]

We sincerely thank the reviewer for the detailed and constructive feedback, and we appreciate your positive comments on our pilot study and the efficiency of our method. Below, we provide additional experiments and clarifications to address each of your questions.

W1: The gains are marginal and lack statistical support.

Ans: Thank you for raising this concern. We added independent training runs for GRPO and CARE under the same training settings as STAC. R3-Table 1 reports the individual results and summary statistics to assess whether the observed gains exceed training variability.

R3-Table 1: Performance across four independent runs (%, Mean ± SD). The reported SD is calculated across the four run-level averages; in the original Table 1, SD is the mean of the six benchmark-level SDs.

Seed GroupMethodPMC-VQAMedXQAOMVQAVQA-RADSLAKEPathVQAAvg.
G1GRPO56.3026.6080.5058.3082.6064.9061.53
CARE56.5026.4080.8058.7082.8065.3061.75
STAC56.4027.0580.9960.0983.3265.3162.19
G2GRPO56.2526.2580.7258.7683.2065.4061.76
CARE56.4525.6080.7558.9882.4865.3761.61
STAC57.4028.2581.0559.4283.6065.1362.48
G3GRPO56.6525.4080.7358.7683.2064.5061.54
CARE56.3525.8080.7359.4282.4865.5061.71
STAC57.0027.7581.3159.8783.0366.1762.52
G4GRPO56.1026.9080.2058.7082.4064.4061.45
CARE56.6526.7080.9159.0083.2064.9061.89
STAC57.1026.9581.4460.3183.0365.8062.44
Mean ± SDGRPO56.3±0.2326.3±0.6580.5±0.2558.6±0.2282.9±0.4164.8±0.4561.6±0.13
CARE56.5±0.1326.1±0.5180.8±0.0859.0±0.3082.7±0.3465.3±0.2661.7±0.12
STAC57.0±0.4227.5±0.6181.2±0.2159.9±0.3983.2±0.2765.6±0.4762.4±0.15

Across these runs, all four STAC runs achieve higher average accuracy than the best observed run of either baseline. Averaging the six benchmark accuracies within each run and applying Welch's test to the four run-level averages per method gives 95% confidence intervals of [+0.59,+1.08][+0.59, +1.08] over GRPO and [+0.43,+0.90][+0.43, +0.90] over CARE. Both intervals exclude zero, supporting a positive average gain across training runs. We will report the multi-run results in the revised Table 1.

Besides, the magnitude of these gains is comparable to that reported in recent medical multimodal RLVR studies [1, 2]. The improvement is also not limited to average accuracy, as shown in Table 10 of Appendix E; STAC further improves Pass@8 and continuation stability, and it does so at modest cost, about 7% additional training time (R3-Table 2) by obtaining its process signal through a single parallel forward pass over short candidate answers, without external evaluators, step-level annotations, or continuation rollouts.

R3-Table 2: Per-step training-time breakdown of GRPO, CARE, and STAC (seconds, Mean ± SD) on Qwen3-VL-4B-Instruct. Only key training stages are listed.

Training StageGRPOCARESTAC
Rollout generation46.24 ± 2.7946.40 ± 2.8146.58 ± 3.08
Candidate preparation & scoring——4.63 ± 1.95
Actor update30.98 ± 7.5231.46 ± 7.5231.51 ± 7.98
Old log-prob computation12.37 ± 2.0612.85 ± 2.0912.91 ± 2.23
Reference model forward11.46 ± 1.9311.96 ± 2.0011.98 ± 2.14
Total time / step103.0104.7109.8
Relative training cost1.00×1.02×1.07×
AuthorsOpenReview ↗

Response to Reviewer VxG5 [2/3]

W2: The prefix-support signal is noisy and uncalibrated.

Ans: Thank you for highlighting this concern. We will address your concern from the following perspectives:

  1. Correctness concordance: Prefix support reflects commitment to the eventual answer, which can be correct or incorrect, explaining the near-chance correctness concordance of 0.51–0.56. STAC therefore conditions the process reward on final-answer correctness, reinforcing high support for correct answers and discouraging it for incorrect ones. Removing this conditioning in the Direct Prefix Support ablation lowers accuracy in Table 2.

  2. Informative stability rankings: Beyond the positive within-query correlations of ρ≈0.36\rho \approx 0.36–0.420.42, the pairwise concordance Cwithin=Pr⁡(Pi>Pj∣Si>Sj)C_{\mathrm{within}}=\Pr(P_i>P_j\mid S_i>S_j) reaches 0.674–0.782, compared with 0.5 under random ranking, as reported in Table 4 in Appendix B. Figure 1b further shows that high-support correct trajectories retain the correct answer more often than low-support ones. The Shuffled Support ablation in Table 2 also supports the value of this trajectory-level ranking signal.

  3. Fallback frequency: Neutral fallback occurs in 10.6%, 8.2%, and 5.5% of groups entering rescaling at early, middle, and late training stages, mainly due to singleton candidate sets. STAC assigns these groups neutral support because their trajectories cannot be distinguished by this score. Details are provided in our response to Q1 below.


Q1: What fraction of training examples trigger the neutral fallback P~=0.5\widetilde{P}=0.5? When it does not trigger, how large is Pmax⁡−Pmin⁡P_{\max}-P_{\min} typically?

Ans: Thank you for this insightful question. We analyzed the neutral fallback behavior across early, middle, and late training stages using 512 queries with 8 rollouts per query at each stage. The fallback and singleton rates below use the groups entering rescaling as their denominator.

R3-Table 3: Neutral fallback frequency and within-group support spread (Pmax⁡−Pmin⁡P_{\max} - P_{\min}). ∣Cg∣=1|\mathcal{C}_g|=1 denotes the proportion of groups with a singleton candidate-answer set.

StageFallback Rate∥Cg∥=1\|\mathcal{C}_g\|=1Non-Fallback Span (Mean ± SD)Median
Early10.6%10.6%0.76 ± 0.180.80
Middle8.2%8.2%0.74 ± 0.190.77
Late5.5%4.9%0.69 ± 0.200.72

As shown in R3-Table 3, neutral fallback is infrequent among groups entering min-max rescaling and comes almost entirely from singleton candidate sets where ∣Cg∣=1|\mathcal{C}_g|=1. In practice, these singleton sets occur when some rollouts in a group fail format validation while the remaining valid rollouts all produce the same correct answer.

By contrast, fallback caused solely by Pmax⁡−Pmin⁡≤ϵP_{\max} - P_{\min} \le \epsilon when ∣Cg∣>1|\mathcal{C}_g| > 1 is nearly absent across training stages. Moreover, non-fallback groups have a large support spread. Both the mean and median of Pmax⁡−Pmin⁡P_{\max} - P_{\min} remain large throughout training, far above the threshold ϵ=10−6\epsilon = 10^{-6}. This confirms that group-wise rescaling acts on well-separated support values rather than amplifying numerical noise.

AuthorsOpenReview ↗

Response to Reviewer VxG5 [3/3]

W3&Q2: Can the authors provide a controlled variant where prefix support is replaced by a terminal-confidence proxy under the same outcome-conditioned modulation?

Ans: We sincerely appreciate this suggestion for a more controlled comparison. To isolate the effect of early prefix support, we evaluated an Outcome-Conditioned Avg-LogP variant that replaces prefix support with the average token log-probability of the generated rollout, while keeping the same group-wise normalization, outcome-conditioned reward modulation, and process-reward weight as STAC:

R3-Table 4: Controlled comparison of different policy-internal signals under the same reward structure. GRPO, CARE, and STAC results are taken from Table 1 of the submitted manuscript.

MethodPMC-VQAMedXQAOMVQAVQA-RADSLAKEPathVQAAvg.
GRPO56.326.680.558.382.664.961.5
CARE56.526.480.858.782.865.361.8
Outcome-Cond. Avg-LogP56.725.780.759.282.664.461.6
STAC57.027.581.259.983.265.662.4

As shown in R3-Table 4, STAC outperforms both CARE and the Avg-LogP variant across all six benchmarks, while the Avg-LogP variant provides little improvement over GRPO and underperforms CARE. While rollout average log-probability mainly reflects token-level generation fluency across the full text, STAC's candidate-relative prefix support measures how strongly an early reasoning state supports the final answer against competing options. This controlled comparison confirms that STAC's gains come from its prefix-support signal rather than outcome conditioning alone. We will add this ablation to Section 5.3 in our latest manuscript.


W4&Q4: Generalization across model scales (7B–8B) and relation to MedVR.

Ans: Many thanks for raising these important questions regarding model scaling and related work. To test how STAC scales to larger models, we extended our Zero-RL experiments to Qwen3-VL-8B-Instruct on the same benchmarks used for the 2B and 4B models:

R3-Table 5: Zero-RL performance across 2B, 4B, and 8B model scales.

ModelMethodPMC-VQAMedXQAOMVQAAvg.
Qwen3-VL-2B-InstructZero-shot48.921.368.846.3
GRPO49.322.269.146.9
STAC49.623.269.947.5
Qwen3-VL-4B-InstructZero-shot53.425.477.952.2
GRPO53.625.878.152.5
STAC54.126.178.753.0
Qwen3-VL-8B-InstructZero-shot53.630.877.453.9
GRPO53.931.177.954.3
STAC54.331.778.654.9
  1. Generalization across model scales: As shown in R3-Table 5, STAC consistently outperforms GRPO across all three model scales and all evaluated benchmarks. On the newly evaluated 8B backbone, STAC achieves the same average margin over GRPO as on the 2B model, showing that the improvement does not shrink as model size increases. Across all three scales in this challenging Zero-RL setting, which operates without SFT warm-up and includes two out-of-distribution benchmarks, MedXQA and OMVQA, STAC consistently doubles the RL gain over the zero-shot model compared with standard GRPO.
  2. Relation to MedVR [2]: MedVR and STAC both use group rollout information to refine credit assignment in medical multimodal RLVR without extra annotations, but they focus on different aspects of reasoning. MedVR uses spatial bounding-box consensus among successful rollouts to supervise visual tool-use actions, whereas STAC uses policy-internal prefix support to measure the continuation stability of individual reasoning trajectories, modulated by terminal correctness. The two methods are therefore complementary, addressing visual grounding consistency and reasoning trajectory stability, respectively.

We will add the 8B results to Table 3 and include an additional discussion of MedVR in the Related Work of our revised manuscript.


References.
[1] Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning. TPAMI 2026.
[2] MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning. ICLR 2026.


Thank you for the detailed feedback. We hope the added analyses and scale results, together with the clarified scope of our claims, address your concerns, and we respectfully ask that you reconsider your rating in light of these revisions. We also welcome further questions.

Reviewer_gAcuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STAC augments GRPO with an early-prefix support reward. The policy scores candidate answers under a sampled reasoning prefix, and terminal correctness determines whether higher or lower support is rewarded. Experiments cover six medical VQA benchmarks, component ablations, and continuation-stability analyses.

Critical strengths
  1. The method does not require a separate evaluator or branching continuation rollouts during training. The pilot study and prefix ablations provide useful evidence for the reward design.

  2. STAC outperforms GRPO and CARE on all six benchmarks in Table 1. Appendix E reports larger Pass@8 gains over GRPO: 4.4 and 5.8 percentage points on the 400-query subsets of MedXpertQA-MM and PMC-VQA, respectively.

Critical weaknesses
  1. The accuracy gains over the closest baselines are limited. In Table 1, STAC reaches 62.4% average accuracy, compared with 61.5% for GRPO and 61.8% for CARE: improvements of 0.9 and 0.6 percentage points. In the Zero-RL setting, Table 3 also shows average gains of 0.6 and 0.5 points over GRPO on the 2B and 4B backbones. Although consistent, these modest gains make the empirical case for the proposed reward refinement less compelling.
  2. Table 1 reports standard deviations over four runs for STAC but no corresponding GRPO or CARE variability, leaving the reliability of these small margins unclear.
Questions for authors

Could the authors provide comparable multi-run results for GRPO and CARE, ideally using matched training seeds, and report the uncertainty in their differences from STAC?

5 comments
AuthorsOpenReview ↗

Response to Reviewer gAcu [1/2]

We sincerely thank the reviewer for the constructive feedback and for recognizing the practical value of our evaluator-free method, the insights from our pilot and ablation studies, and the Pass@8 improvements. Below, we address your comments in detail.


W1: The accuracy gains over GRPO and CARE are relatively modest, including those in the Zero-RL setting.

Ans: We appreciate your attention to the effect size and would like to contextualize these improvements from three perspectives: 1. Gain magnitude in the context of medical RLVR: STAC outperforms GRPO and CARE across all six medical VQA benchmarks in Table 1, with gains comparable in magnitude to those reported in recent medical multimodal RLVR studies [1, 2]. Measured from the SFT starting point in Table 2, STAC also achieves larger RL-stage accuracy gains than GRPO, particularly on MedXQA.
2. Roughly doubled RL-stage gains in Zero-RL: Although the absolute margins are modest, STAC achieves roughly twice GRPO’s accuracy gain over the zero-shot backbone on both the 2B and 4B models in Table 3. These gains are obtained without SFT format warm-up, with evaluation covering both in-domain benchmarks and two out-of-distribution benchmarks, MedXQA and OMVQA. We also run the Zero-RL experiment on 8B models and report the results in R2-Table 1; the same pattern holds at this larger scale, further supporting our method's generalization.
3. Additional exploration and stability gains at low overhead: STAC also improves Pass@8 and continuation stability, as shown in Table 10 in Appendix E, with only 6.6% additional training time reported in R2-Table 2. Its process signal requires a single forward pass over short candidate answers, without external evaluators, step-level annotations, or continuation rollouts.

R2-Table 1: Zero-RL performance across 2B, 4B, and 8B model scales.

ModelMethodPMC-VQAMedXQAOMVQAAvg.
Qwen3-VL-2B-InstructZero-shot48.921.368.846.3
GRPO49.322.269.146.9
STAC49.623.269.947.5
Qwen3-VL-4B-InstructZero-shot53.425.477.952.2
GRPO53.625.878.152.5
STAC54.126.178.753.0
Qwen3-VL-8B-InstructZero-shot53.630.877.453.9
GRPO53.931.177.954.3
STAC54.331.778.654.9

R2-Table 2: Per-step training-time breakdown of GRPO, CARE, and STAC (seconds, Mean ± SD) on Qwen-3VL-4B-Instruct. Only key training stages are listed.

Training StageGRPOCARESTAC
Rollout generation46.24 ± 2.7946.40 ± 2.8146.58 ± 3.08
Candidate preparation & scoring——4.63 ± 1.95
Actor update30.98 ± 7.5231.46 ± 7.5231.51 ± 7.98
Old log-prob computation12.37 ± 2.0612.85 ± 2.0912.91 ± 2.23
Reference model forward11.46 ± 1.9311.96 ± 2.0011.98 ± 2.14
Total time / step103.0104.7109.8
Relative training cost1.00×1.02×1.07×
AuthorsOpenReview ↗

Response to Reviewer gAcu [2/2]

W2&Q1: The lack of multi-run variability for GRPO and CARE makes the reliability of the improvements unclear. Could the authors report the uncertainty in their differences from STAC?

Ans: Thanks for this helpful suggestion. To address your concern, we conducted four independent runs for GRPO and CARE using matched seed groups under the same configuration as STAC, and report the full results below:

R2-Table 3: Performance comparison of GRPO, CARE, and STAC across four matched seed groups (%, Mean ± SD). The reported SD is calculated across the four run-level averages; in the original Table 1, SD is the mean of the six benchmark-level SDs.

Seed GroupMethodPMC-VQAMedXQAOMVQAVQA-RADSLAKEPathVQAAvg.
G1GRPO56.3026.6080.5058.3082.6064.9061.53
CARE56.5026.4080.8058.7082.8065.3061.75
STAC56.4027.0580.9960.0983.3265.3162.19
G2GRPO56.2526.2580.7258.7683.2065.4061.76
CARE56.4525.6080.7558.9882.4865.3761.61
STAC57.4028.2581.0559.4283.6065.1362.48
G3GRPO56.6525.4080.7358.7683.2064.5061.54
CARE56.3525.8080.7359.4282.4865.5061.71
STAC57.0027.7581.3159.8783.0366.1762.52
G4GRPO56.1026.9080.2058.7082.4064.4061.45
CARE56.6526.7080.9159.0083.2064.9061.89
STAC57.1026.9581.4460.3183.0365.8062.44
Mean ± SDGRPO56.3±0.2326.3±0.6580.5±0.2558.6±0.2282.9±0.4164.8±0.4561.6±0.13
CARE56.5±0.1326.1±0.5180.8±0.0859.0±0.3082.7±0.3465.3±0.2661.7±0.12
STAC57.0±0.4227.5±0.6181.2±0.2159.9±0.3983.2±0.2765.6±0.4762.4±0.15

As presented in R2-Table 3, STAC achieves higher mean accuracy than both baselines on every benchmark, and each of the four STAC runs achieves higher average accuracy than the best run of either baseline. We average the six benchmark accuracies within each run and use the four resulting run-level averages per method for Welch's test. The 95% confidence intervals for STAC's mean gain are [+0.59,+1.08][+0.59, +1.08] percentage points over GRPO and [+0.43,+0.90][+0.43, +0.90] over CARE. Both intervals exclude zero, supporting a positive average gain across training runs. We will report the multi-run results in our revised manuscripts.

References [1] Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning. TPAMI 2026. [2] MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning. ICLR 2026.


We appreciate your constructive suggestions. We hope the multi-run results and uncertainty analysis clarify the magnitude and reliability of the gains, and we would be grateful if you could reconsider your rating in light of this new evidence. Thank you once again for your time and thoughtful feedback, and we also welcome any follow-up questions.

Thank you for the additional runs. The matched-seed results and the confidence intervals address my concern about reliability, and I agree the gain over GRPO and CARE is not an artifact of training variance. My concern about effect size remains. With four runs per method, the average margin is about 0.8 points over GRPO and 0.7 over CARE, essentially what the original Table 1 showed. The Zero-RL results, including the new 8B runs, are still single runs with margins of 0.5–0.6 points, and the “doubled RL gain” framing rests on a GRPO gain of only 0.3–0.6 points over the zero-shot model. Given the limited improvement relative to the added training cost, I will keep my score.

Reviewer_ffSROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

STAC aims to improve credit assignment in medical multimodal RLVR by using early reasoning prefixes to assess support for the eventual answer. During training, it scores candidate answers directly under the policy and combines this support with final-answer correctness to modify the trajectory reward. The motivation is sound: process rewards are not easy to obtain (e.g. annotation, external signal), while outcome-only RLVR can reward spurious trajectories that happen to reach the correct answer.

Critical strengths
  • I find the motivation relevant and the approach practically appealing. Obtaining a process-related signal without an additional evaluator, step-level annotations, or continuation sampling during training could be useful.

  • I appreciate the evaluation across six medical VQA benchmarks. The improvements over GRPO and CARE are small but consistent.

  • I find the ablations on prefix position, shuffled support, and outcome conditioning useful for understanding which parts of the method contribute to the results.

Critical weaknesses
  1. My main concern is novelty. The underlying intuition reminds me a lot of look-ahead rollouts used in tree search and test-time scaling to assess the value intermediate states. Related ideas have also been used during training, for example in VinePPO and ProcessThinker. I recognize that STAC uses a cheaper answer-support proxy and conditions its contribution on the observed outcome. However, I am not yet convinced that the paper establishes a sufficiently clear methodological advance over these existing principles. The medical process-supervision discussion could also include process reward agents (ICML 26).

  2. The results seem reasonable, but not super striking / exciting. The improvements over GRPO range around 1-2 percentage points. I appreciate the standard deviations over four STAC runs, but corresponding variability is missing for GRPO and CARE. Given the size of the gains, I am not sure how confidently these differences can be separated from training variability.

  3. I am not fully convinced by the connection between answer stability and sound reasoning. A trajectory can consistently reach the correct answer through a shortcut or an unsupported explanation. Higher continuation stability therefore does not by itself demonstrate better visual grounding or clinically valid deductions. Similarly, lower stability on incorrect trajectories could mean switching to another wrong answer. The claims about robust medical reasoning seem stronger than what these measurements establish.

  4. There seem to be few specification and presentation issues. Eq. 6 and Algorithm 1 omit the process-reward coefficient of 0.2 used in Appendix C. This needs clarification because it changes the balance between correctness and stability. The open-ended verifier also lacks an explicit rule for converting Jaccard similarity into binary correctness. Figure 1 would benefit from a more self-contained caption, and I would explicitly define ∣oi∣|o_i| as the rollout’s token length.

Questions for authors
  1. What is the main methodological distinction from existing prefix-value estimation and rollout-based process rewards? Could the authors provide a comparison against a representative baseline under a matched compute budget?

  2. Could the authors report repeated-run results for GRPO and CARE, together with uncertainty estimates for the performance differences? (Maybe i just didn't find these reported results yet)

  3. Does the reduction in incorrect-answer persistence correspond to more wrong-to-correct transitions? Is there evidence that STAC improves reasoning validity or visual grounding beyond answer consistency?

  4. Which process-reward coefficient was used in the experiments, and how exactly is binary correctness determined for open-ended answers? (Maybe it was not obvious where it is described)

  5. What is the measured training overhead relative to GRPO, including candidate scoring?

7 comments
AuthorsOpenReview ↗

Response to Reviewer ffSR [1/4]

We sincerely thank the reviewer for the detailed and constructive feedback. We are glad that you found our motivation practically relevant, our improvements consistent across six medical VQA benchmarks, and our ablation studies informative. Below we provide point-by-point clarifications and additional experimental results to address your questions.


W1&Q1: The methodological novelty of STAC over existing prefix-value estimation and rollout-based process rewards remains unclear. Please clarify the key distinctions and provide a comparison with a representative baseline under a matched compute budget.

Ans: Thank you for raising this question. We address it from the following two perspectives:

1. Methodological distinction.

While STAC shares the goal of using intermediate states for credit assignment, it differs in both the computation and meaning of its process signal. STAC measures how strongly a reasoning prefix supports the trajectory’s own eventual answer, capturing trajectory continuation stability. It obtains this signal without continuation sampling and combines it with final-answer correctness to guide optimization.

Specifically, VinePPO [1] and ProcessThinker [2] estimate intermediate state values through multiple continuation rollouts, while PRA [3] relies on an external reward agent at inference time. STAC instead compares a prefix’s support for the eventual answer against competing candidates in a single parallel teacher-forced forward pass. As shown in Section 3.2, this support reflects stability rather than correctness, so STAC reinforces stable support for correct answers while penalizing persistent commitment to incorrect ones.

2. Controlled comparison with continuation-rollout RL.

To directly assess the benefit of STAC’s rollout-free signal, we compare GRPO, STAC, and a Continuation-Rollout RL baseline on Qwen3-VL-2B-Instruct under otherwise identical experimental settings. The baseline uses the same prefix position as STAC, samples eight continuations per prefix, and replaces STAC’s prefix-support score PP with the resulting empirical answer stability, while keeping the remaining reward formulation unchanged. We compare all three methods under the comparable GPU-hour budget, allowing their training-step counts to differ according to computational cost.

R1-Table 1: Performance and training costs on Qwen3-VL-2B-Instruct under the matched GPU-hour budget. Accuracy is reported in percent.

MethodPMC-VQAMedXQAMax. Training StepRelative Cost vs. GRPOTime / Step (s)
Zero-shot48.921.3———
GRPO49.222.32851.00×81.8
Continuation-Rollout RL49.322.71651.73×141.5
STAC49.523.452521.10×90.4

Under a matched GPU-hour budget, STAC outperforms both Continuation-Rollout RL and GRPO on PMC-VQA and MedXQA. Its lower per-step overhead also lets it complete more training steps than Continuation-Rollout RL, and thus more policy updates within the same budget. These results show that the prefix-support signal is a more compute-efficient route to process supervision than directly sampling continuation stability.

R1-Table 2: Per-step training-time breakdown on Qwen3-VL-2B-Instruct. Stage times are reported as Mean ± SD in seconds. Only key training stages are listed.

Training StageGRPOSTACContinuation-Rollout RL
Rollout generation31.66 ± 2.1833.14 ± 2.1032.73 ± 2.16
Candidate preparation & scoring—6.81 ± 0.7456.21 ± 6.37
Actor update25.59 ± 5.0125.35 ± 3.2424.20 ± 1.89
Old log-prob computation11.85 ± 1.5011.97 ± 1.0712.38 ± 1.08
Reference model forward10.89 ± 1.4610.89 ± 1.0510.52 ± 0.93
Total time / step81.890.4141.5
Relative cost vs. GRPO1.00×1.10×1.73×

Moreover, the profiling results in R1-Table 2 further quantify this difference. Relative to GRPO, STAC’s training cost is 1.10×, compared with 1.73× for Continuation-Rollout RL. Stage-level measurements show that candidate preparation and scoring take 6.81 seconds for STAC versus 56.21 seconds for the continuation-based baseline. These results support the practical benefit of obtaining the process signal without additional autoregressive continuation sampling.

We will add the controlled comparison and profiling results and expand the related-work discussion in our revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer ffSR [2/4]

W2&Q2: The performance gains over GRPO and CARE are relatively modest, and the lack of repeated-run variability for these baselines makes it unclear whether the observed improvements are statistically distinguishable from normal training fluctuations.

Ans: We sincerely appreciate this helpful suggestion. Following your advice, we ran four independent training runs for both GRPO and CARE under the same training configuration and report their mean and standard deviation alongside STAC in R1-Table 3:

R1-Table 3: Performance comparison of GRPO, CARE, and STAC over four independent runs (%, Mean ± SD). The reported SD is calculated across the four run-level averages; in the original Table 1, SD is the mean of the six benchmark-level SDs.

Seed GroupMethodPMC-VQAMedXQAOMVQAVQA-RADSLAKEPathVQAAvg.
G1GRPO56.3026.6080.5058.3082.6064.9061.53
CARE56.5026.4080.8058.7082.8065.3061.75
STAC56.4027.0580.9960.0983.3265.3162.19
G2GRPO56.2526.2580.7258.7683.2065.4061.76
CARE56.4525.6080.7558.9882.4865.3761.61
STAC57.4028.2581.0559.4283.6065.1362.48
G3GRPO56.6525.4080.7358.7683.2064.5061.54
CARE56.3525.8080.7359.4282.4865.5061.71
STAC57.0027.7581.3159.8783.0366.1762.52
G4GRPO56.1026.9080.2058.7082.4064.4061.45
CARE56.6526.7080.9159.0083.2064.9061.89
STAC57.1026.9581.4460.3183.0365.8062.44
Mean ± SDGRPO56.3±0.2326.3±0.6580.5±0.2558.6±0.2282.9±0.4164.8±0.4561.6±0.13
CARE56.5±0.1326.1±0.5180.8±0.0859.0±0.3082.7±0.3465.3±0.2661.7±0.12
STAC57.0±0.4227.5±0.6181.2±0.2159.9±0.3983.2±0.2765.6±0.4762.4±0.15

We observe that STAC consistently outperforms GRPO and CARE across all six benchmarks. In addition, every individual STAC run achieves a higher overall average accuracy than the best observed run of either GRPO or CARE. To quantify the uncertainty of the performance differences, we first average accuracy over the six benchmarks within each run, then apply Welch's t-test to the four run-level averages per method. The resulting 95% confidence intervals for STAC's mean gain are [+0.59,+1.08][+0.59, +1.08] over GRPO and [+0.43,+0.90][+0.43, +0.90] over CARE. Both intervals exclude zero, supporting a positive average gain across training runs. We will clarify the two SD definitions when updating Table 1 in the revised manuscript.

Regarding the magnitude of the gains, the comparison in Table 1 is strictly controlled all methods start from the same SFT checkpoint and share identical training settings, so the difference isolates the effect of the reward credit-assignment rule alone. Under this setting, the gains are comparable in magnitude to those reported in recent medical multimodal RLVR studies [4, 5]. Moreover, the benefit of STAC extends beyond average accuracy, it improves the Pass@8 exploration ceiling and continuation stability, as shown in Table 10 in Appendix E, without external evaluators, step-level annotations, or continuation rollouts.

AuthorsOpenReview ↗

Response to Reviewer ffSR [3/4]

W3&Q3: It remains unclear whether improved answer stability reflects sound reasoning or visual grounding, and whether reduced incorrect-answer persistence leads to more wrong-to-correct transitions rather than merely switching between incorrect answers.

Ans: Thank you for these insightful comments. We would like to claim that STAC does not reward stability alone, since support and correctness are largely independent (Figure 1c), it conditions prefix support on terminal correctness, favoring stably supported correct trajectories and penalizing persistent commitment to incorrect ones. To test whether this goes beyond answer consistency, we analyze error recovery and reliance on visual evidence:

1. Wrong-to-correct transitions (Tw→cT_{w → c}). To examine where continuations go after an initially incorrect trajectory, we independently sampled 1,000 questions each from MedXQA and PMC-VQA and applied the continuation-probing protocol of Section 5.4. We partition the continuations into three outcomes, keeping the original wrong answer (Tw→wT_{w → w}), switching to the correct answer (Tw→cT_{w → c}), and switching to another wrong answer (Tw→w′T_{w → w'}), and the results are presented below.

R1-Table 4: Continuation outcome decomposition for initially incorrect trajectories on an independently sampled set of 1,000 questions per dataset (%). Values are reported as GRPO / STAC (Δ\Delta = STAC − GRPO, in pp).

DatasetPrefixTw→wT_{w → w} ↓Tw→cT_{w → c} ↑Tw→w′T_{w → w'}
MedXQA25%53.7 / 53.3 (−0.4)15.6 / 15.8 (+0.2)30.8 / 30.9 (+0.1)
50%72.8 / 71.6 (−1.2)8.1 / 8.6 (+0.5)19.1 / 19.8 (+0.7)
75%85.1 / 83.2 (−1.9)4.0 / 4.9 (+0.9)10.9 / 11.9 (+1.0)
PMC-VQA25%60.3 / 58.4 (−1.9)21.4 / 22.4 (+1.0)18.4 / 19.2 (+0.8)
50%78.8 / 77.4 (−1.4)10.7 / 11.0 (+0.3)10.5 / 11.6 (+1.1)
75%89.5 / 88.2 (−1.3)4.9 / 5.8 (+0.9)5.6 / 5.9 (+0.3)

Across all six settings, STAC lowers error persistence Tw→wT_{w → w}, and this reduction is consistently accompanied by higher wrong-to-correct transition rates Tw→cT_{w → c}. We note that part of the reduced persistence also shifts to other wrong answers (Tw→w′T_{w → w'}), so not every abandoned error is converted into a correct answer. Nevertheless, Tw→cT_{w → c} increases in every setting, showing that weakening persistent commitment to incorrect answers opens a measurable path toward the correct one.

2. Counterfactual visual perturbations. To test whether STAC relies more on visual evidence than on language shortcuts, we ran two counterfactual perturbation experiments: Image Swap, which replaces the original image with a randomly mismatched image, and Image Blackout, which masks the visual input completely. We report the accuracy drop between original and perturbed inputs, denoted as ΔSwap\Delta_{\mathrm{Swap}} and ΔBlackout\Delta_{\mathrm{Blackout}}, to measure visual dependence.

R1-Table 5: Visual dependence under counterfactual image perturbations (%). Δ\Delta Swap = Acc. Orig. − Acc. Swap; Δ\Delta Blackout = Acc. Orig. − Acc. Blackout.

DatasetMethodAcc. Orig. ↑Acc. SwapAcc. BlackoutΔ\Delta Swap ↑Δ\Delta Blackout ↑
MedXQAGRPO26.423.623.4+2.8+3.0
STAC27.022.522.2+4.5+4.8
PMC-VQAGRPO56.130.632.9+25.5+23.2
STAC56.728.830.8+27.9+25.9

From R1-Table 5, we notice that STAC reaches higher accuracy on the original images while showing a larger accuracy drop when the visual input is corrupted or removed. Together, these analyses show increased wrong-to-correct transitions and greater dependence on visual input under STAC. We will include both analyses in the revised manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

Dear Area Chair,

We sincerely thank all three reviewers for their time and thoughtful feedback. We appreciate the reviewers' recognition of the practical motivation and efficiency of our evaluator-free approach (ffSR, gAcu, VxG5), the carefully designed pilot study and informative ablations (ffSR, gAcu, VxG5), and the consistent improvements across six medical VQA benchmarks (ffSR, gAcu). These comments, together with the constructive concerns and follow-up discussions, have helped us strengthen our work.

In response, we have conducted extensive additional experiments and analyses, including controlled comparisons under matched compute budgets and training steps, multi-run statistical evaluations, larger-scale experiments, and further studies of reasoning stability and visual dependence. We have also updated the manuscript to incorporate new results and clarify methodological and implementation details, with further evidence in our follow-up responses. Key revisions are summarized below.

Reviewer ConcernRelevant Reviewer(s)Our Actions and Revisions
1. Methodological Novelty and Comparison to Prior WorkffSR, VxG5We clarified that STAC leverages group-relative prefix support for the eventual answer against competing answers from other rollouts, rather than additional look-ahead rollouts or intermediate state-value estimation. We further distinguished STAC from VinePPO, ProcessThinker, and PRA, and discussed its relationship with MedVR.
2. Controlled Comparisons with Continuation-Based MethodsffSR, gAcuWe added matched-GPU-hour and matched-step comparisons with Continuation-Rollout RL. STAC performs better under both settings, with 1.10× GRPO's per-step cost versus 1.73× for the continuation baseline; the matched-step baseline requires more compute.
3. Statistical Reliability of Performance GainsffSR, gAcu, VxG5We conducted four independent runs for GRPO and CARE using matched seed groups. The 95% confidence intervals for STAC's average improvements are [+0.59, +1.08] pp over GRPO and [+0.43, +0.90] pp over CARE. All four STAC runs also outperform the best baseline run in average accuracy.
4. Prefix-Support Reliability and Signal IsolationVxG5We analyzed within-query ranking reliability, neutral fallback frequency, and support spread, confirming informative support differences across training stages. We also introduced an Outcome-Conditioned Avg-LogP baseline under the same reward structure, where STAC achieves 62.4% versus 61.6% average accuracy, isolating the benefit of candidate-relative prefix support over a terminal-confidence proxy.
5. Reasoning Stability and Visual DependenceffSRWe conducted continuation-transition analyses on 1,000 questions each from MedXQA and PMC-VQA, observing increased wrong-to-correct transitions at all evaluated prefix positions. Additional Image Swap and Image Blackout experiments show greater dependence on visual input under STAC, beyond answer consistency alone.
6. Generalization Across Model ScalesgAcu, VxG5We extended Zero-RL evaluations to Qwen3-VL-8B-Instruct, confirming consistent improvements over GRPO across 2B, 4B, and 8B backbones. These Zero-RL experiments show cross-scale applicability.
7. Training Efficiency and Computational OverheadffSR, gAcu, VxG5We provided detailed training-time profiling on both 2B and 4B backbones. STAC incurs approximately 10% and 7% additional training time over GRPO, respectively. Its process signal requires only a single parallel, no-gradient forward pass. We also conducted experiments under a matched compute budget, where STAC still achieves the highest scores.
8. Implementation Details and Presentation ClarityffSRWe clarified the process-reward coefficient (0.2), the open-ended correctness threshold (Jaccard similarity > 0.8), and the definition of rollout length, while refining the equations, algorithm description, and Figure 1 caption.

We believe these experiments and clarifications strengthen the empirical support for STAC and clarify the scope of its methodological contribution. In the follow-up discussion, our matched-seed results and confidence intervals were acknowledged as addressing the concern on statistical reliability.

We respectfully hope that the Area Chair will consider the revised manuscript, the additional experimental evidence, and the full author–reviewer discussion in assessing the contribution and practical value of this work.

We sincerely thank the Area Chair and all reviewers again for their time, constructive feedback, and thoughtful consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.