acceptodds
Under review as a conference paper at ICLR 2027

Improving Generalization Robustness of Multimodal RLVR

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) makes Multimodal Large Language Models (MLLMs) more accurate, but the gains are brittle: simply rewording the instruction or changing the answer template can erase them, which challenges reliable deployment in high-stakes scenarios like medical VQA. The base model barely reacts to such changes, so the brittleness is created by RL post-training itself. We trace it to two issues of the standard RLVR objective. First, the binary verifier conflates format with content, so the reward signal cannot tell a wrong answer apart from a misformatted one. Second, the training prompts cover only a thin slice of the semantically equivalent prompts met at deployment, so a policy that is optimal on the training template can behave differently on unseen ones. Both failures call for an objective over the full class of semantically equivalent prompts, and we identify two conditions that help achieve it: separating format from semantics in the reward, and enforcing answer invariance across equivalent prompts. We therefore propose Prompt-Invariant RLVR (PIRL), consisting of a dynamic trinary reward and a consistency regularizer driven by an embedding-space adversary. Under template stress testing, PIRL's mean accuracy drops by only –%, whereas GRPO drops –. On dynamic evaluation, PIRL also achieves the smallest drop among trained methods.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, MLLMs, Post-training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BcRaOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the prompt robustness problem of Reinforcement Learning with Verifiable Rewards (RLVR) for multimodal large language models. The authors observe that RLVR can significantly improve accuracy on standard prompts but may introduce brittleness when the same task is presented with semantically equivalent prompt variations. They identify two main causes: reward entanglement, where binary verifiers mix formatting correctness and semantic correctness, and prompt dependency, where RLVR only optimizes over a limited distribution of training-time prompt templates. 20285_Improving_Generalization To address this issue, the authors propose Prompt-Invariant RLVR (PIRL), which combines a Dynamic Trinary Reward (DTR), decoupled advantage normalization, embedding-space adversarial prompt perturbation, and consistency regularization to encourage answer invariance across equivalent prompts. 20285_Improving_Generalization Experiments on multiple VQA benchmarks with Qwen2.5-VL-7B and Qwen3-VL-8B demonstrate that PIRL improves robustness under template stress testing compared with GRPO-based baselines.

Critical strengths

Strength 1

The paper provides a strong problem formulation and diagnosis. Instead of simply proposing another RL training heuristic, the authors analyze why RLVR may introduce prompt brittleness and identify reward entanglement and prompt dependency as two fundamental issues. The observation that binary verifiers cannot distinguish formatting failures from semantic failures is particularly insightful because it explains why standard RLVR objectives may lose useful optimization signals.

Strength 2

The proposed PIRL framework provides a meaningful methodological contribution. The Dynamic Trinary Reward and decoupled advantage normalization redesign the reward optimization process by separating semantic correctness from formatting correctness, while the adversarial prompt sampler and consistency regularization provide a practical mechanism for learning invariance across prompt variations. This combination is specifically tailored to the challenges of RLVR training and goes beyond directly applying existing robustness methods.

Strength 3

The experimental evaluation is comprehensive. The authors evaluate PIRL on multiple domains including exam, medical, and legal VQA, using different multimodal backbones. The results consistently show improved robustness under template stress testing compared with GRPO-based baselines. Moreover, the ablation study investigates the contribution of different components, including multi-template training, DTR, and adversarial consistency, which improves confidence in the effectiveness of the proposed framework.

Critical weaknesses

Weakness 1

The main limitation is that the adversarial prompt perturbation mechanism may not fully represent realistic prompt distribution shifts. The method performs perturbations in instruction embedding space, which provides a convenient approximation but may differ significantly from naturally occurring human-written prompt variations. More evaluations using real user prompts, human-generated paraphrases, or LLM-generated diverse instructions would strengthen the robustness claims.

Weakness 2

The proposed semantic reward introduces additional dependencies on external evaluation models. Since DTR uses a frozen DeBERTa-large-MNLI model for semantic scoring, it remains unclear how sensitive PIRL is to the quality of this evaluator and whether the improvements partially come from introducing a stronger reward model rather than the proposed RL framework itself.

Questions for authors

Q1:

How sensitive is PIRL to the choice and quality of the semantic reward model?

Q2:

What is the additional computational cost compared with standard GRPO training?

Q3:

How large does the prompt template pool need to be to achieve stable robustness improvements?

7 comments
AuthorsOpenReview ↗

Response to Reviewer BcRa (1/5): W1

Thank you for your strong support of our problem diagnosis, PIRL's design, and our cross-domain experiments. Below we address your questions regarding natural prompt shifts, external reward-model dependency, compute overhead, and template pool size.

W1: Embedding-space perturbations vs. realistic human-written prompt variations

Response: We agree that continuous perturbations in instruction-embedding space are a local training-time approximation and do not by themselves prove robustness to natural user phrasing. To test realistic distribution shifts directly, we evaluated PIRL on held-out human-written natural-language rewrites.

Evaluation design. We constructed 22 unseen natural-language prompt rewrites across three distinct axes:

  • Wording & syntactic rewrites (16 prompts): 4 simple paraphrases and 12 substantial syntactic/pragmatic restructurings (e.g., I'm stuck on the question above. Can you help me work out the answer? Please put the final answer in \boxed{}.), retaining the \boxed{} delimiter.
  • Output-format shifts (3 prompts): Requesting <final>...</final>, a standalone FINAL RESULT: line, or a JSON object with a single "answer" key.
  • Explanation-requirement shifts (3 prompts): Requesting no explanation, a brief explanation, or explicit evidence from the question or image, retaining \boxed{}.

All prompts were written by human annotators and lightly reviewed/polished with an LLM, modifying only the post-question instruction while keeping the image, question stem, and ground-truth label unchanged. We compare PIRL and GRPO-MT under identical training data, five-template pool, and budget (Qwen3-VL-8B, 32 policy updates, 128 training questions, 2,048 rollouts) across 512 test questions (128 each from MMK12, MathVista, VQA-RAD, and DocVQA).

Results (PIRL − GRPO-MT differences, in percentage points):

Rewrite Axis (# Prompts)Rewrite Mean Acc. (↑)Worst-Template Acc. (↑)Answer Flip Rate (↓)
Wording & syntax (16)+1.2+1.2+1.1+1.1−4.4-4.4
Output format (3)+0.7+0.7+0.2+0.2−2.7-2.7
Explanation constraints (3)+2.1+2.1+2.1+2.1−3.5-3.5

We have three findings:

  1. With nearly identical Standard accuracy (48.6% vs. 48.5%), PIRL improves mean accuracy across all three unseen natural-language shift categories (+1.2, +0.7, and +2.1 pp), extending our evaluation well beyond training-time template swaps.
  2. Worst-template accuracy also improves across all three categories (+1.1, +0.2, and +2.1 pp), showing gains in both average and worst-case prompt performance.
  3. Gains are largest when toggling explanation requirements (+2.1 pp on both mean and worst-template accuracy, and −3.5 pp in answer flips), indicating that PIRL maintains both accuracy and answer consistency when users ask for bare answers versus justified reasoning. We hope these results help address your concern regarding realistic prompt shifts.
AuthorsOpenReview ↗

Response to Reviewer BcRa (2/5): W2

W2: Dependency on frozen DeBERTa-large-MNLI vs. contribution of the PIRL framework itself

Response: Thank you for raising this point, since separating the external evaluator from the RL framework helps clarify where PIRL's gains actually come from.

First, in PIRL, RsR_s is graded first by deterministic rules across three tiers: Exact (all extracted candidates match a∗a^\ast), Redundant (a∗a^\ast appears alongside conflicting candidates), and Partial (numeric candidate within relative tolerance or normalized substring match). Only responses that earn zero rule credit fall through to the frozen DeBERTa-large-MNLI model. Moreover, the NLI model runs only during training-time reward computation (~31s per refresh out of ~1,580s total training time) and is never used at inference or evaluation.

To cleanly separate the external NLI model's contribution from PIRL itself, we trained rule-only PIRL: identical to full PIRL in every respect, except the NLI fallback is removed entirely (responses without rule credit simply receive Rs=0R_s = 0). Evaluated under the exact same setup as W1 (accuracy, %):

MethodSemantic Score RsR_sStandardWording & Syntax (16)Output Format (3)Explanation Constraints (3)
GRPO-MTNone (binary reward)48.547.656.645.4
PIRL (rule-only)Rules only (no NLI)47.348.157.646.3
PIRL (full framework)Rules + DeBERTa-large-MNLI48.648.857.347.5
  1. PIRL's robustness gains do not rely on introducing an external neural reward model: even with DeBERTa completely removed, rule-only PIRL outperforms GRPO-MT across all three unseen rewrite categories (+0.5 to +1.0 pp), showing that the core robustness improvements stem from decoupled reward normalization and adversarial consistency.
  2. Conversely, holding the exact same DeBERTa scorer fixed, replacing PIRL's DAN with a validation-tuned Scalar combination drops wording-rewrite accuracy by 1.8 pp (95% CI [+1.1,+2.6][+1.1, +2.6]; see xGsM W1/W4), confirming that a stronger scorer alone does not deliver PIRL's gains without decoupled normalization.
AuthorsOpenReview ↗

Response to Reviewer BcRa (3/5): Q1

Q1: Sensitivity of PIRL to the choice and quality of the semantic reward model

Response: Building on W2, we ran two controlled ablations on the semantic scorer: (1) removing the NLI model entirely (rule-only PIRL, retaining rule-based content credit), and (2) replacing DeBERTa-large-MNLI with RoBERTa-large-MNLI to test sensitivity to NLI model family and quality.

All training and evaluation settings match W1/W2; columns other than Standard report the equal-weight macro-average across the three rewrite categories (%):

Method / Semantic ScorerStandard Acc. ↑Rewrite Macro-Mean Acc. ↑Worst-Template Acc. ↑Answer Flip Rate ↓Answer Extractability ↑
PIRL (rule-only, no NLI)47.350.745.810.395.9
PIRL (rule + RoBERTa-large-MNLI)47.150.245.410.496.1
PIRL (rule + DeBERTa-large-MNLI)48.651.245.710.995.0
Baseline: GRPO-MT (no RsR_s)48.549.944.514.492.2
  1. Scorer choice causes modest shifts in accuracy point estimates, with no single variant dominating every metric. Default DeBERTa achieves the highest Standard (48.6%) and rewrite macro-average accuracy (51.2%, beating rule-only by +0.5 pp and RoBERTa by +1.0 pp), whereas rule-only and RoBERTa achieve slightly lower flip rates and slightly higher answer extractability.
  2. Every PIRL variant, with or without an NLI model, substantially improves prompt stability over GRPO-MT, cutting answer-flip rates from 14.4% down to 10.3%–10.9% and raising worst-template accuracy from 44.5% to 45.4%–45.8%. PIRL is therefore robust to the specific choice (or complete absence) of the external NLI evaluator.
Reviewer_xGsMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies prompt sensitivity introduced or amplified by multimodal RLVR. It proposes PIRL, combining a decomposed reward, separate advantage normalization, adversarial instruction-embedding perturbations. and consistency regularizations.

Experiments show improve template robustness over GRPO and multi-templatge GRPO. The empirical results are promising, particularly the answer-flip diagnostics, but the theoretical and causal claims exceed what is established.

Critical strengths
  1. Relevant problems. The paper highlights the improvements in default-prompt accuracy can accompany increased sensitivity to instruction working and output requirements.
  2. GRPO-MT uses the same training-template pool as PIRL, helping distinguish the full method’s benefits from template augmentation alone.
  3. Lower answer-flip rates and higher worst-template accuracy on several benchmarks provide stronger evidence than average accuracy changes alone.
Critical weaknesses
  1. Limited methodological novelty. PIRL combines several established ingredients: its trinary reward is equivalent to additive format and correctness rewards up to a constant; DAN closely follows GDPO’s reward-wise normalization and subsequent batch normalization; and latent adversarial training with consistency regularization has substantial precedent, including LAP and SMART. The contribution therefore rests primarily on their adaptation and integration into multimodal RLVR. While this integration may be useful, the current theory and ablations do not sufficiently establish the distinctive benefit of the proposed design over straightforward combinations of existing methods. The authors should delineate the new technical contributions more precisely and include stronger comparisons that isolate their incremental value.
  2. The theory does not establish the proposed causal explanation or an E2E guarantee. The non-identifiability result shows that training objectives leave unseen behavior unconstrained, but does not explain why GRPO selects brittle policies.
  3. Preserving reward variance does not establish useful semantic supervision. The variance-floor result assumes sufficient within-bucket variation in the semantic score. The paper does not demonstrate how consistently this assumption holds or whether the heuristic/NLI score reliably ranks semantic correctness. These measurements are needed to connect the theory to the observed gains.
  4. Component attribution remains inconclusive. Several individual ablation differences fall below the paper’s stated uncertainty threshold. ADV bundles adversarial search with consistency regularization, leaving their separate contributions unclear.
  5. The robustness claims require qualification. Near-zero signed average degradation can conceal answer changes and cancellation across benchmarks. The base model already exhibits substantial answer-flip rates, so the evidence supports amplification of existing sensitivity. T-Stress also mixes wording, formatting, and restrictions on explanations, while leaving medical yes/no instructions unchanged.
Questions for authors
  1. Can you measure within-bucket semantic-score variance during training and audit whether the score’s rankings agree with human judgments of answer correctness?
  2. Can you separate adversarial search from consistency regularization, compare against random perturbations, and include a tuned scalar-reward baseline with paired confidence intervals?
11 comments
AuthorsOpenReview ↗

Response to Reviewer xGsM (1/8): W1

We sincerely thank you for reviewing our work and for recognizing the practical relevance of the prompt-sensitivity problem. Following your comments and questions, we have sharpened our positioning and theoretical scope, added a format-gating gradient analysis and rollout-log audit, measured within-bucket semantic variance alongside a double-blind human ranking audit, conducted new ablation experiments with question-paired bootstrap confidence intervals, and added evaluations across unsigned gaps, perturbation axes, and open-ended medical subsets.

W1: Precedents for trinary reward, DAN, and adversarial consistency; isolating incremental contributions over straightforward combinations

W1-1: Explaining technical contributions and why straightforward combinations fall short in multimodal RLVR

Thank you for laying out the related literature so clearly. Your summary of the intellectual lineage is spot on, and we explicitly cite and discuss each of these connections in the paper: RtR_t equals an R1-style additive "format + accuracy" reward up to a constant shift (§3.1); DAN's component-wise normalization builds on GDPO (cited in §3.1/Appendix G.4); and instruction-embedding perturbations with consistency regularization share roots with LAP and SMART (§1, §5). We do not claim these three individual ingredients as standalone inventions.

Rather, the straightforward combinations you mention are precisely what our theoretical analysis examines. Specifically, we show that without satisfying both (C1) Reward Decomposition and (C2) Invariance Control, naive combinations in multimodal RLVR leave specific failure modes unresolved:

CombinationWhat Breaks in Multimodal RLVRTheoretical Result
Binary + format bonus (RtR_t, R1-style additive reward)Adds zero within-format reward variance and lowers its share of total variance; all three joint buckets remain flat internally, leaving format-failing responses unranked by content quality.Cor. B.12
Joint group normalization of Rt+αRsR_t + \alpha R_s (Scalar combination)In rollout groups with non-constant RtR_t, a small α\alpha suppresses within-bucket semantic signal: at G=16,α=0.01G = 16, \alpha = 0.01 (Remark B.11), the within-bucket share is at most 4.45×10−4sG4.45 \times 10^{-4} s_{\mathcal{G}}, whereas DAN (αadv=1\alpha_{\mathrm{adv}} = 1) retains at least sG/4s_{\mathcal{G}} / 4. (When RtR_t is constant and RsR_s varies, both rank by RsR_s and this suppression bound does not apply.)Thm. B.10, Remark B.11
Prompt-independent dense semantic score RsR_s aloneLeaves objective-level non-identifiability on unseen formats intact.Prop. B.9
Consistency regularization alone (LAP/SMART style)Blind to degenerate policies that ignore instructions and emit one fixed format (Lcons≡0\mathcal{L}_{\mathrm{cons}} \equiv 0); consistency must be paired with a format-sensitive task reward over a template pool (RtR_t).Prop. B.14(iv), (ii)

Tailoring adversarial consistency to RLVR also introduces two concrete technical departures from LAP and SMART: (1) because verifiable RL rewards are non-differentiable, our adversary estimates ascent directions via a leave-one-out score-function estimator with no pathwise gradient term (Lemma B.17(a)), whereas LAP and SMART differentiate smooth supervised losses; and (2) our consistency KL is computed strictly between clean and perturbed prompts sharing the same delimiter (§3.2), preventing conflicts with cross-template format compliance.

W1-2: Controlled comparison isolating DAN against a validation-tuned scalar baseline

To directly test the straightforward combination of joint normalization (Rt+αRsR_t + \alpha R_s) + adversarial perturbations + consistency, we ran a budget-matched comparison on Qwen3-VL-8B (fixed training data, five-template pool, 32 policy updates; full 7-way ablation in W4 & Q2). For the Scalar baseline, we swept α∈0.001,0.01,0.1,1,10\alpha \in \\{0.001, 0.01, 0.1, 1, 10\\} on a held-out validation set, selected the best weight (α=0.001\alpha = 0.001), and evaluated it on the 512-item P-Stress benchmark. DAN keeps αadv=1\alpha_{\mathrm{adv}} = 1 with no hyperparameter sweep. Both runs share the exact same data, template pool, semantic scorer, adversary, and consistency weight, differing only in how the two reward terms enter the policy advantage:

MethodP-Stress Mean Acc. ↑Worst-Template Acc. ↑Answer Flip Rate ↓
DAN (PIRL, fixed αadv=1\alpha_{\mathrm{adv}} = 1)48.843.29.3
Tuned Scalar (α=0.001\alpha = 0.001)47.041.110.8
DAN − Scalar (pp)+1.8+1.8+2.1+2.1−1.5-1.5

Without tuning its mixing weight, DAN outperforms the validation-tuned Scalar baseline across all three metrics (+1.8 pp mean accuracy, +2.1 pp worst-template accuracy, and −1.5 pp flip rate), isolating the practical value of decoupled normalization over a tuned linear combination.

AuthorsOpenReview ↗

Response to Reviewer xGsM (2/8): W2-1

W2: Non-identifiability does not explain why GRPO selects brittle policies, nor provide an E2E guarantee

We agree with this critique. Prop. B.6 establishes an objective-level blind spot: unseen formats are left unconstrained by the training objective. This result does not by itself prove why GRPO converges toward brittle policies. We have narrowed our framing of Prop. B.6 accordingly. Below, we complement it with a format-gating gradient analysis and training-log diagnostics that show how binary verifiers choke off content learning on low-compliance prompts, and clarify the exact scope of our theoretical bounds.

W2-1: Binary rewards gate content-learning updates by the format-compliance rate

Fix a prompt and denote format compliance by pf=Pr⁡[Vf=1]p_f = \Pr[V_f = 1] and content correctness by pc=Pr⁡[Vc=1]p_c = \Pr[V_c = 1]. Under the binary verifier V=VfVcV = V_f V_c, the mean reward satisfies

μ=Pr⁡[Vf=1,Vc=1]≤pf.\mu = \Pr[V_f = 1, V_c = 1] \le p_f.

Consider a parameter bb that shifts only the content-correctness probability pc=σ(b)p_c = \sigma(b) while keeping the conditional response laws given VcV_c fixed, so that

∂blog⁡πθ=Vc−pc.\partial_b \log \pi_\theta = V_c - p_c.

Under population normalization (ϵnorm=0\epsilon_{\mathrm{norm}} = 0), using Cov⁡(V,Vc)=μ(1−pc)\operatorname{Cov}(V, V_c) = \mu(1 - p_c), the expected reward-driven update along the content coordinate bb is

gbbin=E[Norm⁡(V)(Vc−pc)]=(1−pc)μ1−μ≤(1−pc)pf1−pf.g_b^{\mathrm{bin}} = \mathbb{E}\big[\operatorname{Norm}(V)(V_c - p_c)\big] = (1 - p_c)\sqrt{\frac{\mu}{1 - \mu}} \le (1 - p_c)\sqrt{\frac{p_f}{1 - p_f}}.

Notably, this bound requires no independence assumption between VfV_f and VcV_c. For any fixed pcp_c, taking pf→0p_f \to 0 drives the content update gbbin→0g_b^{\mathrm{bin}} \to 0: even when responses contain the right answer, missing the requested format zeroes out the binary verifier's ability to reinforce correct content.

In a simplified regime where Vf⊥VcV_f \perp V_c, if two templates for the same question share the same content accuracy pcp_c, whichever template already has higher format compliance pfp_f receives a stronger content-learning signal. Under independent per-template content parameters and equal sampling/step sizes, pre-existing format disparities induce unequal local content-learning speeds (in a shared-parameter MLLM, net trajectory changes also depend on cross-template gradient interference).

Under explicit conditions, DAN preserves a semantic update component that does not vanish with pfp_f. Let

Z=Norm⁡(Rt)+αadvNorm⁡(Rs)Z = \operatorname{Norm}(R_t) + \alpha_{\mathrm{adv}} \operatorname{Norm}(R_s)

denote the pre-batch-normalization DAN signal. If Vf⊥VcV_f \perp V_c, the conditional law of RsR_s given VcV_c is independent of VfV_f, and ρs=Corr⁡(Rs,Vc)>0\rho_s = \operatorname{Corr}(R_s, V_c) > 0, then

E[Z(Vc−pc)]≥αadvρspc(1−pc).\mathbb{E}\big[Z(V_c - p_c)\big] \ge \alpha_{\mathrm{adv}} \rho_s \sqrt{p_c(1 - p_c)}.

Holding the joint law of (Rs,Vc)(R_s, V_c) fixed, this positive content signal stays bounded away from zero even as pf→0p_f \to 0. The condition ρs>0\rho_s > 0 also makes clear why your audit in W3/Q1 is essential: within-bucket variance makes responses distinguishable, while positive alignment between RsR_s and true correctness determines whether that preserved signal points in a useful direction.

AuthorsOpenReview ↗

Response to Reviewer xGsM (3/8): W2-2 and W2-3

W2-2: Training-log evidence that DTR + DAN preserves learning signal in failure groups where binary GRPO collapses to zero

We audited the rollout logs from our Qwen3-VL-8B controlled runs (32 policy updates, 128 rollout groups of G=16G = 16 per run, 2,048 rollouts total) and counted how often groups without a fully correct+formatted response still retain non-zero policy advantages:

Training Setup (# Runs Logged)Groups w/ Zero Fully Valid Rollouts (V=0V = 0 for all 16)...of Which Retain Non-Zero AdvantageGroups w/ 100% Format Failure (Vf=0V_f = 0 for all 16)...of Which Retain Non-Zero Advantage
GRPO-MT, binary reward (3 runs)1350 (0.0%)580 (0.0%)
DTR + DAN, logged runs (15 runs)621579 (93.2%)218204 (93.6%)

(Note: The DTR + DAN row aggregates all 15 logged DTR + DAN training runs from our study, a broader log set than the 7 evaluation configs in W4.)

In GRPO-MT, all 135 zero-success groups receive identically zero advantage. Yet 116 of those 135 groups have non-constant RsR_s, and 20 of the 58 all-format-failure groups already contain responses graded Exact on content by our rule tier. Binary verification throws all of that signal away.

Conversely, among the 621 zero-success groups encountered during DTR + DAN training, 579 (93.2%) retain non-zero advantages (all 579 would collapse to zero under a binary verifier). Among the 218 all-format-failure groups, 204 (93.6%) retain non-zero advantages; the remaining 14 have constant RsR_s, matching the degenerate case noted in Remark B.11. These logs confirm the core gating mechanism in practice, while whether non-zero advantages translate into test gains additionally depends on score alignment and downstream optimization [1].

W2-3: Scope of theoretical guarantees

We fully agree that our analysis does not constitute an end-to-end finite-step convergence guarantee to test accuracy. Our results characterize three specific levels:

  • Answer invariance on the training pool: Prop. B.14(iii) bounds average pairwise answer-marginal TV across the pool by twice the trinary pool suboptimality ζ\zeta:

Eτ1,τ2∼Ptrain[TV⁡(Pτ1π,Pτ2π)]≤2ζ,\mathbb{E}_{\tau_1, \tau_2 \sim P_{\mathrm{train}}}\big[\operatorname{TV}(P^\pi_{\tau_1}, P^\pi_{\tau_2})\big] \le 2\zeta,

which bounds pairwise expected reward differences for any common answer-level reward by 2ΔRζ2\Delta_R \zeta (Prop. B.14(vii)).

  • Local drift across same-delimiter prompts: Under the current perturbed policy law PϕP^\phi, Prop. B.14(v) bounds the answer-marginal KL by the expected sum of token KLs: D_{\mathrm{KL}}(A_\\#P^\phi\\|A_\\#P^0) \le \mathbb{E}_{P^\phi}[|y|\ell_\theta(y)] \le L_{\max}\mathbb{E}_{P^\phi}[\ell_\theta(y)], where AA is the shared answer extractor and ℓθ(y)\ell_\theta(y) is the length-averaged token KL in Eq. (8).
  • Transfer to unseen templates in the same grader class: Prop. B.16, Lemma B.17, and Cor. B.18 provide transfer bounds conditioned on template distance and a pathwise upper bound ω\omega on local worst-case drops. Finite-step adversarial drops lower-bound local sensitivity (providing a diagnostic rather than certifying ω\omega), and unseen delimiter families outside the pool remain unconstrained (Prop. B.6).

We have updated the manuscript to separate these claims explicitly: non-identifiability characterizes coverage blind spots in the objective, whereas format-gating analysis and training logs explain how content-learning signals are lost under binary rewards and preserved under DTR + DAN.

[1] Harshay Shah et al., "The Pitfalls of Simplicity Bias in Neural Networks," NeurIPS 2020.

Reviewer_MPbsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates how minor changes in instruction wording or answer formats can erase RLVR performance gains, attributing this to binary verifiers that conflate format with content and limited coverage of semantically equivalent prompts during training. To address these issues, the authors propose Prompt-Invariant RLVR (PIRL), which combines a dynamic trinary reward with consistency regularization driven by an embedding-space adversary. Experiments across diverse settings demonstrate its effectiveness.

Critical strengths
  1. Clear motivation and problem formulation. The paper identifies a practically relevant robustness gap in RLVR-trained MLLMs, where accuracy gains on standard prompts may not persist under semantically equivalent template variations.
  2. Diverse experimental settings and comprehensive evaluation. The experiments cover exam, medical, and legal/document VQA, using two Qwen-VL backbones and multiple evaluation protocols.
Critical weaknesses
  1. The method implemented by the authors does not fully align with their motivation. The authors aim to address the entanglement between formatting and content in RLVR, but their actual method penalizes prediction discrepancies between clean and perturbed prompts. The theoretical objective is insufficient to fully establish that optimizing the proposed objective can disentangle output templates from semantics. Moreover, the paper does not adequately define semantic consistency.
  2. The authors do not sufficiently disentangle the functions and contributions of the individual components. In the ablation results in Table 10, the numerical results of different ablation configurations are relatively close across multiple metrics. Consequently, it remains unclear which specific components account for the performance gains and how the components interact to produce the combined improvements.
Questions for authors
  1. Could the authors provide representative case studies illustrating how DTR and DAN distinguish formatting failures from semantic errors, and how adversarial training with consistency regularization changes model behavior across prompts, especially cases in medical and legal settings?
  2. The current evaluation primarily focuses on relatively minor variations in answer formats and instruction templates. How well does PIRL generalize to more complex prompt transformations that preserve task semantics but substantially change the wording, syntactic structure, or way the task is expressed?
8 comments
AuthorsOpenReview ↗

Response to Reviewer MPbs (1/6): W1 (Beginning)

We sincerely thank you for reviewing our work and for recognizing the clear motivation, problem formulation, and comprehensive cross-domain evaluation across exam, medical, and legal/document VQA. Following your two concerns and two questions, we have clarified the theoretical alignment and formal definition of answer-level semantic consistency, added a 7-configuration controlled ablation isolating individual components and their interactions, compiled diagnostic case studies in medical and document VQA, and evaluated PIRL on unseen syntactic and task-framing natural-language rewrites. Our specific responses are as below.

W1: Alignment between method and motivation, definition of semantic consistency, and whether the theory proves disentanglement

Thank you for reading the theoretical analysis so closely. We would first like to clarify a potential misreading caused by space constraints in the main text: our paper diagnoses two distinct failure modes reward entanglement (§2.2) and prompt dependency (§2.3). The two parts of PIRL target them separately, each backed by specific theoretical guarantees explaining why it is needed and what it ensures. Format–content entanglement is handled on the reward side by DTR and DAN, whereas the consistency term penalizing clean/perturbed discrepancies targets prompt dependency. We first summarize this mapping below and then address each aspect in four sub-points:

Diagnosed IssueCorresponding PIRL ComponentTheoretical ResultsScope of Validity
Reward entanglement: Binary verifiers assign the same zero score to format failures, wrong answers, and joint failuresDTR (RtR_t, RsR_s) + DANLemma B.1, Prop. B.2, Cor. B.12, Thm. B.10, Prop. B.13Holds under each result's stated assumptions; the pfp_f-free positive floor in Thm. B.10 relies on the within-bucket dispersion condition in (C1). Empirical laws of finite rollout groups are covered in Remark B.11.
Prompt dependency: Under restricted template coverage, the objective is non-identifiable on unseen templatesRtR_t over the template pool + consistency loss under adversarial perturbationsProps. B.6, B.9, B.14, B.16; Lemma B.17; Cor. B.18Population/optimum level of the objective; the same-delimiter drift bound constrains answer distributions on paired prompts; transfer bounds are conditioned on a pathwise local worst-case drop bound, with realized adversarial drops serving as a sensitivity diagnostic.
AuthorsOpenReview ↗

Response to Reviewer MPbs (2/6): W1-1 and W1-2

W1-1: Format–content entanglement is resolved on the reward side, and our theory shows why it must be addressed there

  • Where binary rewards lose content signal. For a binary verifier, the within-format share of reward variance is Φwf=πW/pf\Phi_{\mathrm{wf}} = \pi_W / p_f (Lemma B.1, where πW\pi_W is the formatted fraction among zero-reward rollouts and pfp_f is the format-compliance rate). When the policy knows the answer but misses the requested delimiter, this share becomes tiny, and multi-template training plus adversarial perturbations actively create such prompts (Prop. B.2). Binary verifiers discard content signal precisely where robustness training needs it most.
  • Adding a format reward alone does not recover lost content distinctions. Adding an R1-style format bonus (an additive "format + accuracy" reward, which equals RtR_t up to a constant shift) does not increase within-format reward variance and actually reduces its share of total variance, i.e., Φwf(Rt)<Φwf(V)\Phi_{\mathrm{wf}}(R_t) < \Phi_{\mathrm{wf}}(V) (Cor. B.12). While formatted responses are still separated into right vs. wrong, RtR_t remains constant within each of the three joint buckets defined by format compliance and extracted correctness. In particular, it still cannot tell apart content quality among format-failing rollouts.
  • DTR + DAN provides a floor independent of the format-compliance rate. Under the within-bucket dispersion condition E[Var⁡(Rs∣B)]≥m\mathbb{E}[\operatorname{Var}(R_s \mid B)] \ge m in (C1), the within-format share of the DAN signal satisfies Φwf(Z)≥4αadv2m/(1+αadv)2\Phi_{\mathrm{wf}}(Z) \ge 4\alpha_{\mathrm{adv}}^2 m / (1 + \alpha_{\mathrm{adv}})^2 (at least mm when αadv=1\alpha_{\mathrm{adv}} = 1), free of both pfp_f and the reward-mixing weight αrew\alpha_{\mathrm{rew}} (Thm. B.10). Under the standardized finite-group setup of Remark B.11, for any realized rollout group of size G=16G = 16 with non-constant RtR_t, setting αrew=0.01\alpha_{\mathrm{rew}} = 0.01 and αadv=1\alpha_{\mathrm{adv}} = 1 gives DAN a within-joint-bucket signal share of at least sG/4s_{\mathcal{G}} / 4, whereas single-group normalization of Rt+αrewRsR_t + \alpha_{\mathrm{rew}} R_s retains at most 4.45×10−4sG4.45 \times 10^{-4} s_{\mathcal{G}}. (When RtR_t is constant within a group and RsR_s varies, both scalar normalization and DAN rank responses by RsR_s, so this suppression bound does not apply to those groups.) Prop. B.13 gives the dynamic-weighting view of DAN: rollouts inside the same joint bucket are ordered by RsR_s, so a format miss no longer erases content distinctions captured by the semantic score.

When the abstract refers to separating format from semantics in the reward, it refers precisely to this signal-level decoupling: format errors no longer wipe out content signal, and content distinctions are not diluted by pfp_f. Empirically, within-bucket dispersion holds broadly in the failure buckets of Thm. B.10: in our new training logs, 88.7% of format-failure groups and 76.2% of wrong-answer groups exhibit non-zero RsR_s variance (see our response to Reviewer xGsM, W3).

W1-2: The consistency term targets prompt dependency

When training templates cover only a subset of formats, two policies that are equally optimal on the training distribution can differ by mBΔRm_B \Delta_R on an unseen format (Prop. B.6), and adding a prompt-independent dense semantic score leaves this blind spot intact (Prop. B.9). Fixing the reward alone is therefore not enough for prompt invariance. Our consistency loss compares only clean/perturbed prompt pairs that share the same delimiter, penalizing answer drift under meaning-preserving rewordings (Prop. B.14(v) upper-bounds the answer-marginal KL) without forcing identical surface outputs across different format instructions.

AuthorsOpenReview ↗

Response to Reviewer MPbs (3/6): W1-3 and W1-4

W1-3: The theory guarantees output-level separation, not representation disentanglement

We do not claim representation-level disentanglement (nor does the main text use the word "disentangle"). Instead, our results characterize an output-level property: surface formatting adapts to the instruction while the extracted answer distribution stays invariant across templates (§2.4: "(C2) asks for invariance of the answer, not of the surface format"). Specifically:

  • If the policy class contains a policy whose delimiter-extracted answer is almost surely correct on every template in the pool, then every RtR_t-optimal policy shares this property, eliminating the answer-level non-identifiability of Prop. B.6 across the pool (Prop. B.14(i)). Moreover, letting ζ=1−Eτ∼Ptrain,,y∼π(⋅∣τ(x))[Rt(y,τ(x))]\zeta = 1 - \mathbb{E}_{\tau \sim P_{\mathrm{train}},\\, y \sim \pi(\cdot \mid \tau(x))}[R_t(y, \tau(x))] denote the trinary suboptimality on the pool, any policy satisfies the TV form of (C2) with ε=2ζ\varepsilon = 2\zeta (Prop. B.14(iii)); thus, smaller suboptimality on the pool directly tightens the bound on average answer-distribution drift.
  • A policy cannot buy answer invariance by ignoring format instructions: outputting a single fixed format across our five-template pool loses at least 1.2 out of 2 reward units (Prop. B.14(ii), Remark B.15). Because consistency alone is blind to such instruction-ignoring policies (Prop. B.14(iv)), it must be paired with RtR_t.
  • For unseen templates within the same grader class, Prop. B.16, Lemma B.17, and Cor. B.18 bound transfer degradation in terms of template distance and an upper bound on the maximum local reward drop along connecting paths. Adversarial search with a finite number of steps provides a lower bound on this local drop, making it useful as a diagnostic of prompt sensitivity. However, it does not by itself certify the required upper bound along the entire path. Delimiter families outside the pool remain a blind spot (Prop. B.14(vi)(b)).

These results characterize the objective's optima and advantage signals rather than end-to-end finite-step training convergence. In the revision, we have added the summary table above to §2.4 with explicit scope notes, and removed phrasing that could be misread as representation disentanglement (such as "leaves visual understanding underdeveloped" in the Introduction).

W1-4: Formal definition of semantic consistency

While the original submission does not use the phrase "semantic consistency," Definition A.2 of that submission ("(C2) Invariance Control (answer-level)"; now Definition 2.1) is its exact formalization: §2.3 defines the semantic equivalence class [![x]!]=τ(x):τ∈T[\\![x]\\!] = \\{\tau(x) : \tau \in \mathcal{T}\\} over semantics-preserving transformations T\mathcal{T}, §2.1 specifies that transformed prompts inherit the ground-truth answer a∗(x)a^\ast(x), and this definition requires the delimiter-extracted answer marginals P^\theta_\tau = (\mathcal{A}_\tau)_\\# \pi_\theta(\cdot \mid \tau(x)) to remain close across templates:

Eτ1,τ2∼ρx[D(Pτ1θ,∣,Pτ2θ)]≤ε.\mathbb{E}_{\tau_1, \tau_2 \sim \rho_x}\big[D(P^\theta_{\tau_1} \\,\\|\\, P^\theta_{\tau_2})\big] \le \varepsilon.

Overall, when the image, task meaning, and ground-truth answer are unchanged, the final answer distribution extracted under each instruction's delimiter should remain consistent, and answer-flip rate and worst-template accuracy are its direct empirical counterparts. We agree that giving (C2) only inline in the main text while deferring the original Definition A.2 to the appendix made this harder to follow. We have moved it into the main text as Definition 2.1, explicitly termed it answer-level semantic consistency, and noted that all invariance guarantees are defined relative to the task transformation family T\mathcal{T}.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief Response to the Area Chair and Reviewers

Dear Area Chair and Reviewers,

Thank you for your thoughtful and detailed reviews. We appreciate Reviewer MPbs's recognition of our motivation and cross-domain evaluation, Reviewer xGsM's positive assessment of the answer-flip and worst-template accuracy diagnostics, and Reviewer BcRa's encouraging feedback on our problem formulation and the method design. We have revised the manuscript in light of your feedback to make clearer what our theoretical and empirical results establish.

Our core goal in this work is to study the gap between standard-prompt accuracy gains and prompt robustness in the multimodal RLVR, and to address issues of reward entanglement and limited training-prompt coverage within a unified objective. Below, we clarify the role of each proposed module, the exact scope of our theoretical results, and what our empirical evidence supports. We also present new experiments including auditing the semantic scorer, isolating individual components, and stress-testing under broader natural-language prompt rewrites.

Best regards,

The Authors of PIRL

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.