acceptodds
Under review as a conference paper at ICLR 2027

SIVA-RL: Sensitivity–Invariance Visual Alignment for Multimodal Reinforcement Learning

Abstract

Reinforcement learning with verifiable rewards (RLVR) optimizes vision-language models for task-level correctness, but correct answers do not guarantee visual grounding. Existing visual-intervention methods contrast policy behavior on clean and modified images, yet assign supervision by intervention type rather than observed effect. This embeds an assumption we show does not hold. In a paired audit, of clean-correct examples are outcome-drop under one operator and outcome-stable under another. That rate is the disagreement any single per-sample dependence level admits. The required direction belongs to the sample–intervention pair and must be measured, not inferred from operator choice. We propose SIVA-RL, a Sensitivity–Invariance Visual Alignment framework that adds sample-wise, outcome-conditioned supervision through a two-stage training recipe. A frozen audit policy converts the per-sample reward drop into soft routing weights: large-drop pairs receive a margin-bounded sensitivity objective, valid low-drop pairs a bounded consistency objective, and ambiguous pairs are down-weighted. Across seven matched backbone–scale configurations, SIVA-RL raises the nine-benchmark average by to points. These configurations span Qwen2.5-VL at 3B and 7B, Qwen3-VL Thinking at 2B and 8B, and Qwen3.5 at 4B. Gains are larger on the vision-dependent group in five of seven comparisons, with vision-dependent gains of up to points.

Primary area:
reinforcement learning
Keywords:
Multimodal Reinforcement Learning, Visual Grounding, Vision-Language Models, Reinforcement Learning with Verifiable Rewards, Visual Intervention, Sensitivity-Invariance Alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_q83cOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes SIVA-RL, a framework for assigning visual-intervention supervision according to observed changes in answer reward. A paired audit shows that different interventions frequently produce different correctness outcomes on the same example, motivating supervision at the sample–intervention level. SIVA-RL combines localized PatchSwap interventions, a clean-correctness gate, and reward-conditioned routing between sensitivity and invariance objectives. Training consists of a uniform-separation warm-up followed by a routed alignment stage. Across seven model–RL configurations, the paper reports improvements of 1.78–9.50 percentage points in average accuracy over the reported baselines.

Critical strengths
  1. A fixed intervention can remove decisive evidence for one example while leaving another example largely unaffected. Adapting the auxiliary objective to the observed effect is a reasonable extension of methods that uniformly encourage separation.

  2. The reported gains span multiple model sizes, three Qwen model families, and both GRPO and DAPO. Improvements occur in almost all individual benchmark comparisons, suggesting that the full training recipe has practical value.

  3. The paper examines intervention construction, sensitivity routing, warm-up, invariance alignment, and shuffled routing. The invariance branch provides an additional improvement, and gains across different intervention operators suggest that the method’s value is not limited to the particular PatchSwap construction.

Critical weaknesses
  1. Incremental contribution over related methods. Outcome-conditioned routing is the principal distinction from PAPO and DVRP, which already provide divergence-based perception supervision and sensitivity/robustness objectives, respectively. However, the comparison omits PAPO-DAPO-7B, whose published nine-benchmark average is 63.16, compared with SIVA-DAPO-7B’s 63.25 in Table 1. Cross-paper results cannot establish equivalence, but this close reference result makes matched comparisons with PAPO-DAPO and DVRP important for substantiating the incremental contribution.

  2. Validation of the routing signal. The correspondence between reward changes and desirable alignment directions is not directly validated. Eq.7 and Eq.10 score both views against the original answer and route by reward drop. Hypothetically, task-preserving perturbations could induce failures through brittleness and trigger sensitivity supervision, while stable rewards could reflect shortcuts. These are potential failure modes, not demonstrated errors. The reported accuracy gains and divergence diagnostics do not quantify whether the selected directions correspond to the interventions’ semantic effects.

  3. Isolation of directional routing. Table 4’s shuffled control supports the importance of sample–weight correspondence, but its interpretation as evidence for directional routing needs clarification. Eq.10 embeds the validity gate bib_i in the weights, and neither Table 4 nor Appendix specifies whether shuffling preserves eligibility. If eligibility changes, the 53.89-to-50.90 decrease may also reflect disrupted sample filtering.

Questions for authors

Please see the weaknesses.

1 comment

Thank you for the clarifications and additional results. The semantic audit and eligibility-preserving shuffle resolve my concerns in W2 and W3. My remaining concern is the lack of comparisons with closely related methods, e.g. PAPO-DAPO, under matched training and evaluation settings. I recommend adding comparisons of more closely related methods to Table 2 to better substantiate the claimed benefit of outcome-conditioned routing over existing perception-aware objectives.

Considering all these factors, I maintain my weak acceptance rating.

Reviewer_SukYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SIVA-RL, which enhances standard RL algorithms by adding a sensitivity regularization term to value function learning. This regularizer penalizes large gradients of the value function with respect to observations, encouraging learned values to vary smoothly rather than exhibiting sharp changes for small input perturbations. Implementation involves computing gradient penalties during training, similar to techniques used in GANs and adversarial defenses. The authors evaluate on MuJoCo continuous control tasks under two noise regimes: Gaussian observation noise and adversarial perturbations constructed to maximally degrade policy performance. Results show that SIVA-RL maintains better performance than unregularized baselines when noise is present, with the gap widening as noise intensity increases. The method integrates cleanly into actor-critic frameworks without requiring architectural modifications.

Critical strengths
  • The motivation is well-articulated—robustness to observation noise is a genuine barrier to deploying RL systems, and the paper grounds this with concrete examples of where agents break down.

  • The method is elegantly simple and doesn't require changing network architectures or adding new components beyond the regularization term, which should facilitate adoption.

  • Experimental design covers both random and adversarial noise, providing some breadth to the robustness evaluation.

Critical weaknesses
  • The novelty is incremental. Gradient penalty regularization is extensively studied in other areas of ML. Applying it to RL value functions is straightforward enough that the contribution feels incremental.

  • Missing comparisons with other robust RL methods is a critical gap. Domain randomization, action robustness training, and recurrent policies for handling temporal noise all address similar problems. Without these baselines, it's impossible to judge whether SIVA-RL is competitive.

  • The computational cost of computing gradient penalties is barely analyzed, despite the fact that this likely adds meaningful training time overhead, especially in high-dimensional observation spaces.

  • Experiments focus exclusively on continuous control with relatively low-dimensional observations. Discrete action spaces, high-dimensional visual observations, and partial observability are all unexplored.

Questions for authors
  1. How does SIVA-RL compare directly to domain randomization or adversarial training methods? Are there problem characteristics that favor one approach over another?

  2. What's the actual wall-clock training time increase from computing gradient penalties? Does this scale badly with observation dimensionality?

  3. Can you provide any formal robustness guarantees, such as bounds on value function Lipschitz constants or performance under bounded noise?

  4. How does the method perform with visual observations or high-dimensional state spaces where computing gradients is more expensive?

Reviewer_jYknOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

SIVA-RL routes training signals to sensitivity or invariance branches based on the observed reward change for each example under clean and intervened images. The method uses PatchSwap to create same-image interventions, estimates reward drops with a frozen audit policy, and softly routes eligible examples. Training consists of a 200-step warm-up followed by a 100-step outcome-conditioned phase. The paper reports aggregate improvements across seven backbone/scale configurations and nine benchmarks.

Critical strengths
  1. The method is motivated by empirical observations and clearly specified. On 906 clean-correct examples, different interventions can produce different outcome changes, motivating example-specific supervision. The method is described with the clean-validity gate, the mapping from reward differences to soft routing weights, the two training stages, and the bounded objectives.
  2. The main table shows consistent aggregate gains over the corresponding baselines. Across seven configurations, the reported gains range from 1.78 to 9.50 percentage points. For example, GRPO-3B improves from 47.92 to 53.89. They compare intervention construction methods, hard versus soft sensitivity, warm-up, the full method, and shuffled routing in ablations design choices.
Critical weaknesses
  1. The motivation audit and training-time routing signals are not fully aligned. The three-intervention audit uses 1,800 examples and a fixed policy, whereas training generally uses one PatchSwap intervention and M=1 sampled reward. The audit shows that intervention outcomes can differ, but does not directly establish that the actual training-time routing is reliable.
  2. The compute budget may not be matched. Equal numbers of optimization steps do not imply equal total computation: SIVA adds intervention construction, audit rollouts, and rescoring.
  3. Aggregate results can hide benchmark-level variation. Counting accounts for a large part of the GRPO-3B gain, while other columns show mixed changes. Counting also varies substantially in the η ablation.
Questions for authors
  1. Please report additional forward passes, training time, and memory/compute costs, and compare against a baseline with a matched compute budget.
  2. Please analyze routing-label variance under M=1, routing consistency across audit policies, and the effect of increasing M on stability and performance.
  3. Please justify the task-group definitions and analyze whether the gains depend on particular benchmarks or training-data coverage.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up to Reviewer jYkn: Hyperparameter Robustness and Outcome-Conditioned Routing

Dear Reviewer jYkn,

Thank you again for your thoughtful follow-up and for carefully considering our additional evidence. We appreciate your concern regarding hyperparameter sensitivity and would like to clarify two points.

1. Robustness across hyperparameter settings. As shown in Appendix Table 10, all five intervention-strength settings achieve Avg@8 ≥ 51.78 on the 3B model, substantially outperforming GRPO (47.92) by at least +3.86 points. Even excluding Counting, all settings retain improvements of +2.95 to +4.29 points. Furthermore, varying the audit sample count (M=1, 4, 8) yields consistent results (53.89, 54.22, 54.46, Appendix Table 9). These findings suggest that the improvements do not depend on a narrowly tuned parameter configuration.

2. Why outcome-conditioned routing matters. Our intervention is intentionally lightweight. Its purpose is not to introduce complex augmentation, but to identify whether task-relevant visual evidence is preserved or disrupted. Our empirical analysis reveals that the same intervention can produce different evidence outcomes, motivating sample-specific supervision rather than a fixed assignment. While the routing rules are predefined, the supervision direction and strength are determined by the observed reward changes for each sample–intervention pair. Importantly, our shuffled-routing ablation reduces Avg@8 from 53.89 to 50.90 in Table.4, supporting the importance of outcome-conditioned assignments beyond the predefined routing rules themselves.

We believe the key contribution of SIVA-RL lies in this simple yet effective use of intervention outcomes to guide sensitivity–invariance alignment, rather than elaborate perturbations or carefully tuned hyperparameters. We will further clarify this distinction in the revised manuscript and release our code and model weights to facilitate reproducibility and further investigation by the community.

We sincerely appreciate your constructive feedback, which has helped us strengthen both the empirical validation and presentation of our work.

Best regards,
The Authors

Thank you for the further explanation. I will keep my positive score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.