- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the prompt robustness problem of Reinforcement Learning with Verifiable Rewards (RLVR) for multimodal large language models. The authors observe that RLVR can significantly improve accuracy on standard prompts but may introduce brittleness when the same task is presented with semantically equivalent prompt variations. They identify two main causes: reward entanglement, where binary verifiers mix formatting correctness and semantic correctness, and prompt dependency, where RLVR only optimizes over a limited distribution of training-time prompt templates. 20285_Improving_Generalization To address this issue, the authors propose Prompt-Invariant RLVR (PIRL), which combines a Dynamic Trinary Reward (DTR), decoupled advantage normalization, embedding-space adversarial prompt perturbation, and consistency regularization to encourage answer invariance across equivalent prompts. 20285_Improving_Generalization Experiments on multiple VQA benchmarks with Qwen2.5-VL-7B and Qwen3-VL-8B demonstrate that PIRL improves robustness under template stress testing compared with GRPO-based baselines.
Critical strengths
Strength 1
The paper provides a strong problem formulation and diagnosis. Instead of simply proposing another RL training heuristic, the authors analyze why RLVR may introduce prompt brittleness and identify reward entanglement and prompt dependency as two fundamental issues. The observation that binary verifiers cannot distinguish formatting failures from semantic failures is particularly insightful because it explains why standard RLVR objectives may lose useful optimization signals.
Strength 2
The proposed PIRL framework provides a meaningful methodological contribution. The Dynamic Trinary Reward and decoupled advantage normalization redesign the reward optimization process by separating semantic correctness from formatting correctness, while the adversarial prompt sampler and consistency regularization provide a practical mechanism for learning invariance across prompt variations. This combination is specifically tailored to the challenges of RLVR training and goes beyond directly applying existing robustness methods.
Strength 3
The experimental evaluation is comprehensive. The authors evaluate PIRL on multiple domains including exam, medical, and legal VQA, using different multimodal backbones. The results consistently show improved robustness under template stress testing compared with GRPO-based baselines. Moreover, the ablation study investigates the contribution of different components, including multi-template training, DTR, and adversarial consistency, which improves confidence in the effectiveness of the proposed framework.
Critical weaknesses
Weakness 1
The main limitation is that the adversarial prompt perturbation mechanism may not fully represent realistic prompt distribution shifts. The method performs perturbations in instruction embedding space, which provides a convenient approximation but may differ significantly from naturally occurring human-written prompt variations. More evaluations using real user prompts, human-generated paraphrases, or LLM-generated diverse instructions would strengthen the robustness claims.
Weakness 2
The proposed semantic reward introduces additional dependencies on external evaluation models. Since DTR uses a frozen DeBERTa-large-MNLI model for semantic scoring, it remains unclear how sensitive PIRL is to the quality of this evaluator and whether the improvements partially come from introducing a stronger reward model rather than the proposed RL framework itself.
Questions for authors
Q1:
How sensitive is PIRL to the choice and quality of the semantic reward model?
Q2:
What is the additional computational cost compared with standard GRPO training?
Q3:
How large does the prompt template pool need to be to achieve stable robustness improvements?
7 comments
Response to Reviewer BcRa (2/5): W2
W2: Dependency on frozen DeBERTa-large-MNLI vs. contribution of the PIRL framework itself
Response: Thank you for raising this point, since separating the external evaluator from the RL framework helps clarify where PIRL's gains actually come from.
First, in PIRL, is graded first by deterministic rules across three tiers: Exact (all extracted candidates match ), Redundant ( appears alongside conflicting candidates), and Partial (numeric candidate within relative tolerance or normalized substring match). Only responses that earn zero rule credit fall through to the frozen DeBERTa-large-MNLI model. Moreover, the NLI model runs only during training-time reward computation (~31s per refresh out of ~1,580s total training time) and is never used at inference or evaluation.
To cleanly separate the external NLI model's contribution from PIRL itself, we trained rule-only PIRL: identical to full PIRL in every respect, except the NLI fallback is removed entirely (responses without rule credit simply receive ). Evaluated under the exact same setup as W1 (accuracy, %):
| Method | Semantic Score | Standard | Wording & Syntax (16) | Output Format (3) | Explanation Constraints (3) |
|---|---|---|---|---|---|
| GRPO-MT | None (binary reward) | 48.5 | 47.6 | 56.6 | 45.4 |
| PIRL (rule-only) | Rules only (no NLI) | 47.3 | 48.1 | 57.6 | 46.3 |
| PIRL (full framework) | Rules + DeBERTa-large-MNLI | 48.6 | 48.8 | 57.3 | 47.5 |
- PIRL's robustness gains do not rely on introducing an external neural reward model: even with DeBERTa completely removed, rule-only PIRL outperforms GRPO-MT across all three unseen rewrite categories (+0.5 to +1.0 pp), showing that the core robustness improvements stem from decoupled reward normalization and adversarial consistency.
- Conversely, holding the exact same DeBERTa scorer fixed, replacing PIRL's DAN with a validation-tuned Scalar combination drops wording-rewrite accuracy by 1.8 pp (95% CI ; see xGsM W1/W4), confirming that a stronger scorer alone does not deliver PIRL's gains without decoupled normalization.
Response to Reviewer BcRa (3/5): Q1
Q1: Sensitivity of PIRL to the choice and quality of the semantic reward model
Response: Building on W2, we ran two controlled ablations on the semantic scorer: (1) removing the NLI model entirely (rule-only PIRL, retaining rule-based content credit), and (2) replacing DeBERTa-large-MNLI with RoBERTa-large-MNLI to test sensitivity to NLI model family and quality.
All training and evaluation settings match W1/W2; columns other than Standard report the equal-weight macro-average across the three rewrite categories (%):
| Method / Semantic Scorer | Standard Acc. ↑ | Rewrite Macro-Mean Acc. ↑ | Worst-Template Acc. ↑ | Answer Flip Rate ↓ | Answer Extractability ↑ |
|---|---|---|---|---|---|
| PIRL (rule-only, no NLI) | 47.3 | 50.7 | 45.8 | 10.3 | 95.9 |
| PIRL (rule + RoBERTa-large-MNLI) | 47.1 | 50.2 | 45.4 | 10.4 | 96.1 |
| PIRL (rule + DeBERTa-large-MNLI) | 48.6 | 51.2 | 45.7 | 10.9 | 95.0 |
| Baseline: GRPO-MT (no ) | 48.5 | 49.9 | 44.5 | 14.4 | 92.2 |
- Scorer choice causes modest shifts in accuracy point estimates, with no single variant dominating every metric. Default DeBERTa achieves the highest Standard (48.6%) and rewrite macro-average accuracy (51.2%, beating rule-only by +0.5 pp and RoBERTa by +1.0 pp), whereas rule-only and RoBERTa achieve slightly lower flip rates and slightly higher answer extractability.
- Every PIRL variant, with or without an NLI model, substantially improves prompt stability over GRPO-MT, cutting answer-flip rates from 14.4% down to 10.3%–10.9% and raising worst-template accuracy from 44.5% to 45.4%–45.8%. PIRL is therefore robust to the specific choice (or complete absence) of the external NLI evaluator.
Response to Reviewer BcRa (1/5): W1
Thank you for your strong support of our problem diagnosis, PIRL's design, and our cross-domain experiments. Below we address your questions regarding natural prompt shifts, external reward-model dependency, compute overhead, and template pool size.
W1: Embedding-space perturbations vs. realistic human-written prompt variations
Response: We agree that continuous perturbations in instruction-embedding space are a local training-time approximation and do not by themselves prove robustness to natural user phrasing. To test realistic distribution shifts directly, we evaluated PIRL on held-out human-written natural-language rewrites.
Evaluation design. We constructed 22 unseen natural-language prompt rewrites across three distinct axes:
I'm stuck on the question above. Can you help me work out the answer? Please put the final answer in \boxed{}.), retaining the\boxed{}delimiter.<final>...</final>, a standaloneFINAL RESULT:line, or a JSON object with a single"answer"key.\boxed{}.All prompts were written by human annotators and lightly reviewed/polished with an LLM, modifying only the post-question instruction while keeping the image, question stem, and ground-truth label unchanged. We compare PIRL and GRPO-MT under identical training data, five-template pool, and budget (Qwen3-VL-8B, 32 policy updates, 128 training questions, 2,048 rollouts) across 512 test questions (128 each from MMK12, MathVista, VQA-RAD, and DocVQA).
Results (PIRL − GRPO-MT differences, in percentage points):
We have three findings: