acceptodds
Under review as a conference paper at ICLR 2027

PASA: Post-Merge Perception-Reasoning Asymmetry as a Self-Alignment Signal for MLLMs

Abstract

Model merging offers a direct way to transfer specialized reasoning capabilities to multimodal large language models (MLLMs), but successful transfer must also preserve their ability to ground answers in visual evidence. We find that merging a reasoning-specialized language model into an MLLM improves reasoning performance while degrading fine-grained visual perception, a phenomenon we term post-merge perception–reasoning asymmetry. We investigate whether the reasoning capability retained after merging can itself provide supervision for recovering visual perception. Comparisons on the same self-alignment queries show that reasoning-prompted responses are more accurate than perception-prompted responses, suggesting a source of supervision within the model. Building on this observation, we introduce PASA, a two-stage framework for post-merge self-alignment. PASA first learns to generate structured visual-evidence traces through supervised fine-tuning on synthetic data. It then applies asymmetric GRPO: a reasoning-prompted role supplies detached answer anchors, while perception-prompted trajectories are optimized for answer agreement and structural validity. Both roles share parameters, allowing anchors to be regenerated from the updated model without ground-truth answer labels during reinforcement learning. Across model scales and merging methods, PASA recovers much of the post-merge perception loss while further improving reasoning. Evaluations of generated visual evidence show improved grounding, and stage-wise self-training comparisons support the benefit of asymmetric supervision. These results demonstrate how capabilities retained after merging can serve as an internal learning signal for post-merge self-alignment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Perception–Reasoning Asymmetry, Self-Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_72GbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the perception–reasoning asymmetry arising from merging reasoning-specialized LLMs into multimodal large language models (MLLMs). The authors observe that model merging can improve reasoning performance while degrading fine-grained visual perception. To address this issue, they propose PASA, a two-stage post-merge alignment framework. In the first stage, the merged model is supervised to generate structured visual evidence through SFT. In the second stage, the authors introduce asymmetric GRPO, where reasoning-prompted responses provide pseudo-answer supervision for perception-prompted trajectories. Experimental results across different model scales and merging strategies demonstrate consistent improvements on multiple multimodal benchmarks.

Critical strengths
  1. The framework is intuitive. PASA leverages different prompted behaviors of the same model to construct a self-training signal. The asymmetric supervision strategy is intuitive and does not require a separate teacher model during the RL stage.
  2. The empirical improvements are considerable. The experiments cover multiple model scales (2B, 8B, and 14B) and different merging strategies. PASA achieves consistent improvements over direct merging across several multimodal benchmarks, including MMMU, MathVista, MMStar, and OCRBench.
Critical weaknesses
  1. The proposed method is motivated by the observation that the reasoning-prompted role achieves substantially higher answer accuracy than the perception-prompted role. However, this comparison is conducted after the structured SFT stage, which has already modified the model's perception-oriented behavior. It is therefore unclear whether the observed gap is a consequence of model merging or is primarily induced by the different prompting strategies and the structured SFT stage.
  2. The paper categorizes several multimodal benchmarks into reasoning-oriented and perception-oriented groups, although these benchmarks generally involve both capabilities. Their performance differences do not necessarily establish a clean separation between reasoning improvement and perception degradation.
  3. The effectiveness of asymmetric supervision may be strongly coupled with the specifically designed structured SFT stage. One particularly interesting result is that PASA outperforms gold-answer GRPO after the structured SFT stage, achieving 69.5 versus 66.7 on MathVista. However, the reported results suggest that this advantage may be closely related to the specific SFT data construction rather than asymmetric supervision alone. In particular, Table 3a reports that reasoning-anchor GRPO without Stage I achieves 65.1 on MathVista, slightly below the 65.5 achieved by gold-answer GRPO without Stage I in Table 2. After introducing structured SFT, however, the corresponding scores become 69.5 and 66.7, respectively. This suggests a potentially important interaction between the structured SFT stage and the subsequent asymmetric RL procedure, although the current experiments do not fully isolate this interaction. The structured SFT stage trains the model on synthetic visual-evidence trajectories following a specific schema, thereby modifying the distribution of perception-prompted responses. It is possible that this initialization particularly benefits the subsequent reasoning-anchor supervision, for example, by making the pseudo-answers more compatible with the model's current output distribution or providing more informative reward variations during GRPO. However, the paper does not sufficiently investigate these possibilities.
Questions for authors

See weakness

5 comments

We thank Reviewer 72Gb for the constructive review and for appreciating our paper's soundness, intuitive parameter-shared cross-role design, and consistent gains across model scales and merge strategies. We address all three concerns below.

Does the role gap arise from model merging rather than prompting and Stage-I SFT? (W1)

Yes, the positive reasoning-over-perception role gap is directly caused by cross-modal merging prior to any SFT. Evaluating both roles (PreasonP_{\text{reason}} and PpercP_{\text{perc}}) on the exact same 6,000 queries across {Unmerged Source, Merged} × {Before, After Stage-I SFT} isolates each factor cleanly:

Checkpoint (2B, 6k Pool)MergedStage IReasoning Acc.Perception Acc.Gap (R−P\text{R}-\text{P})
Source MLLMNoNo54.7%69.6%−14.9
Direct MergeYesNo73.9%42.5%+31.4
Source + Stage INoYes57.0%71.2%−14.2
Merge + Stage I (θstab\theta_{\text{stab}})YesYes75.3%47.2%+28.1
Full PASAYesYes83.4%77.5%+5.9
  1. Prompting and Stage-I SFT alone never create a positive R>P\text{R} > \text{P} gap. On the unmerged Source MLLM, perception exceeds reasoning by 14.9 points (69.6% vs. 54.7%). Applying the identical 20,000-trace Stage-I SFT (Source + Stage I) improves both roles slightly while leaving the gap virtually unchanged at −14.2 (71.2% vs. 57.0%), matching Table 2b (Source + Stage I: 55.2 / 64.2 vs. PASA: 59.2 / 69.5 on MMMU / MathVista).
  2. Model merging reverses the role gap before SFT. Directly merging the models (Direct Merge) immediately raises reasoning accuracy (+19.2) while lowering perception accuracy (−27.1), reversing the gap by +46.3 points (−14.9 →\to +31.4 before SFT; +28.1 after SFT).

Evidence of perception degradation and recovery beyond benchmark grouping (W2)

Yes, perception degradation after merging and recovery under PASA hold directly on fine-grained, task-decoupled measurements without relying on benchmark grouping (which Section 4.1 and Appendix A.2 use only as an operational summary):

  1. Exact text, document, and formula reading on OCRBench (Tables 1 & 2a). OCRBench uses deterministic string/exact matching with zero LLM judge, and deductive reasoning cannot guess unseen characters or digits in an image. OCRBench drops after merging (82.5 →\to 79.4 at 2B, 82.9 →\to 76.2 at 8B, 82.9 →\to 78.9 at 14B) and recovers under PASA (83.2, 83.3, 83.7), surpassing Gold GRPO (82.0±0.05) and Majority (82.3±0.21) [1].

  2. Same-image, same-question perception-role recovery ([W1]). Holding the image and question identical across 6,000 queries, merging drops Pperc\mathcal{P}_{\text{perc}} accuracy (69.6% →\to 42.5% pre-SFT; 71.2% →\to 47.2% post-SFT), and Stage-II PASA restores it to 77.5% (+30.3).

  3. Blank-image visual-dependence test. Replacing input images with a blank white canvas drops PASA's accuracy from 69.5% →\to 9.6% on MathVista (−59.9) and 56.6% →\to 5.7% on MMStar (−50.9), confirming direct reliance on visual evidence.

  4. Answer-blind visual-evidence audit (Figure 3a & Table 6). Scoring only image-observable facts in answer-stripped MathVista traces raises the Visual Grounding Score from 64.7 (Direct Merge) →\to 68.2 (Stage I) →\to 75.9 (Full PASA).

Isolating the Stage-I × Stage-II interaction and validating both mechanisms (W3)

Yes. Stage I increases Reasoning-Anchor GRPO's advantage over Gold-Answer GRPO across all five benchmarks (positive relative-to-Gold interaction IR−G=(YesR−YesG)−(NoR−NoG)I_{R-G} = (\text{Yes}_R-\text{Yes}_G)-(\text{No}_R-\text{No}_G) throughout) because Stage I stabilizes parsable structured trajectories while Stage-II ara_r supplies a distributionally reachable target with denser within-group agreement variance.

  1. Robustness across Stage-I SFT variants (Table 3a–c, MMMU / MathVista / MMStar / Hallusion). Zeroing SFT loss on final-answer tokens (Answer-Masked Stage I + PASA) reaches 59.0 / 69.5 / 56.5 / 48.2 vs. default 59.2 / 69.5 / 56.6 / 48.1, preserving the full 69.5 MathVista score without terminal-answer SFT supervision. Replacing external Qwen3-VL-Max traces with parser-filtered self-generated traces from the merged model itself still reaches 58.7 / 69.0 / 56.4 / 47.8 (69.0 vs. 66.8±0.17 for Gold GRPO), while structured traces outperform unstructured traces (65.3 / 46.7 vs. 63.9 / 45.2 on MathVista / Hallusion, Table 3a).

  2. Complete 2×4 Stage-I × Stage-II comparison and rollout diagnostics. Note also that Gold-answer GRPO uses privileged reference answers (a⋆a^\star) during RL, whereas PASA uses zero ground-truth RL labels (Majority [1] is the matched label-free RL baseline). With Stage I, PASA surpasses even privileged Gold GRPO across all five benchmarks:

Stage IStage-II TargetMMMUMathVistaMMStarOCRBenchHallusion
NoNone (Direct Merge)56.763.451.279.445.5
NoMajority (amaja_{\text{maj}}) [1]57.366.053.180.246.1
NoGold (a⋆a^\star)58.265.553.480.246.4
NoReasoning (ara_r)57.765.153.480.947.0
YesNone (Stage I)57.465.353.780.546.7
YesMajority (amaja_{\text{maj}}) [1]57.866.753.982.146.5
YesGold (a⋆a^\star)58.866.754.381.946.9
YesReasoning (ara_r, PASA)59.269.556.683.248.1
LiftStage I on ara_r+1.5+4.4+3.2+2.3+1.1
InteractionIR−GI_{R-G} (vs. Gold)+0.9+3.2+2.3+0.6+0.6

Matched 3-seed means (mean ± sample SD, n=3n=3) for the three With-Stage-I RL rows are 57.8±0.20 / 66.4±0.19 / 54.0±0.10 / 82.3±0.21 / 46.4±0.16 (Majority), 58.7±0.13 / 66.8±0.17 / 54.3±0.14 / 82.0±0.05 / 47.1±0.11 (Gold), and 59.2±0.06 / 69.4±0.21 / 56.4±0.25 / 83.2±0.09 / 48.1±0.15 (PASA), likewise giving positive IR−GI_{R-G} of +1.0 / +3.0 / +2.1 / +0.5 / +0.4 across all five benchmarks.

Rollout Metric (K=8K=8)No Stage I + ara_rStage I + MajorityStage I + GoldFull PASA
Format valid23.8%98.4%98.0%98.6%
Target hit (≧1/8≧ 1/8)66.7%74.0%69.0%77.0%
Agreement variance >0>052.0%67.0%62.0%71.0%

These diagnostics directly confirm both mechanisms you proposed:

  • Without Stage I (65.1), Direct Merge satisfies the structured <caption>...</caption> + \boxed{} format on only 23.8% of rollouts (vs. 98.6% after Stage I), leaving only 52.0% nonzero agreement-variance groups as updates are spent fixing output syntax.
  • With Stage I (69.5 vs. 66.7, +3.2 IR−GI_{R-G}), once structured trajectories are parsable (98.0%–98.6%), external gold answers a⋆a^\star still miss all K=8K=8 rollouts more frequently on hard visual-math items (69.0% hit; 62.0% nonzero agreement-variance groups), whereas same-model ara_r comes from a much stronger role than Majority (75.3% reasoning vs. 47.2% greedy perception) and is distributionally compatible with the shared policy (77.0% hit; 71.0% nonzero agreement-variance groups) while co-evolving upward (75.3% →\to 83.4%). We will add this 2×4 table and rollout analysis to Section 4.4 and streamline Section 3's role-update presentation for clarity.

[1] Wei et al., Unsupervised Post-Training for Multi-Modal LLM Reasoning via GRPO, 2025.

Dear Reviewer,

Thank you again for your thoughtful review and for the time you have spent evaluating our paper.

We have carefully addressed the concerns raised in your review and hope that our rebuttal has helped clarify the issues you pointed out.

As the discussion period is coming to an end, we would like to kindly ask if you have any remaining questions or if there is anything further we can clarify. We would be more than happy to provide additional explanations.

Thank you very much for your time and consideration.

Reviewer_tFCROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper identifies post-merge perception–reasoning asymmetry: merging a reasoning-specialized LLM into an MLLM improves reasoning benchmarks while degrading fine-grained visual perception. Since the merged model's reasoning role answers the same queries far more accurately than its perception role (75.3% vs. 47.2%), the authors propose PASA, a two-stage self-alignment framework: (1) SFT on structured six-field visual-evidence traces, then (2) an asymmetric GRPO in which a detached, greedy reasoning-role answer anchors rewards for sampled perception-role trajectories (answer agreement + format validity), requiring no gold answers during RL. Evaluated across 2B/8B/14B models, three merge recipes, and five benchmarks, PASA recovers much of the lost perception while further improving reasoning; an answer-blind grounding audit and matched supervision ablations (vs. gold-GRPO and MM-UPT) support the cross-role design.

The problem framing and the cross-role self-alignment mechanism are novel, clear, and genuinely valuable, and the validation design is unusually broad and honestly caveated. However, the quantitative claims are currently critically under-verified: single-run point estimates without significance, incomplete reproducibility records, a reward that never checks visual facts, non-negligible anchor error, and a reliance on a model judge for the grounding claim.

Critical strengths
  1. A novel, well-motivated problem and a clean conceptual mechanism. Characterizing post-merge perception–reasoning asymmetry and then using the retained reasoning role as an internal supervisory signal is a fresh and intellectually appealing contribution. The cross-role, detached-anchor construction cleanly separates the target source from the optimized behavior and requires no gold answers or external teacher at RL time—this advances beyond gold-label RL and symmetric self-training (MM-UPT) in a principled way, and the fixed-Stage-I matched comparisons lend it face validity.
  2. Broad, systematic validation with honest limits. The evaluation spans model scales, merge recipes, staged ablations, a model-independent claim about grounding (answer-blind VGS audit), and anchor-reliability. The authors repeatedly flag their own caveats (no causal claim, no fact verification, mismatch risk), which supports credibility and outlines exactly what future work must test.
Critical weaknesses
  1. Reproducibility and statistical verification are currently insufficient for the claims. The paper repeatedly notes that exact run replication requires per-run configs, seeds, exact prompts/parsers, dedup manifests, and evaluation metadata that are "not fully captured in the retained record." The comparisons are single-run point estimates with no significance tests or CIs against baselines, and the only multi-run statistics (PASA only) are reported as aggregates. Because the decisive margins are small (e.g., ~1–2 pts on several metrics, Stage-I vs. masked differences ≤0.2 pts), the reader cannot verify whether the observed ordering is reliable.
  2. The perception-recovery claim rests on feedback that never verifies vision. The RL reward is answer-agreement + format-validity, which cannot detect a caption that is structurally valid but visually fabricated; the independently audited VGS comes from a single model judge with acknowledged imperfect blinding. Combined with ~17%/12% wrong anchors and 32%/13% wrong-anchor following, there is a real risk that parts of the gain reflect a strengthened speaker prior (agreeing with the reasoning role's answer conventions) rather than genuinely improved visual extraction. The paper partially acknowledges this but does not yet rule it out with direct, judge-free evidence (e.g., verifiable probe tasks on the recovered perception role).
Questions for authors

Clarification:

  • Can you release the code, exact prompts/parsers, per-run logs, seeds, and per-config benchmark scores, and report matched significance tests or confidence intervals for PASA vs. gold-GRPO and PASA vs. MM-UPT under the fixed-Stage-I protocol? Without these, the sub-2-point margins cannot be assessed.
  • Why does the reasoning-anchor GRPO beat gold-answer GRPO? You state target correctness alone cannot explain the ordering; what reward-density or trajectory-behavior diagnostics support your interpretation?
5 comments

We thank Reviewer tFCR for the thoughtful feedback and for highlighting the novelty of our problem framing, our detached-anchor cross-role mechanism, and our systematic validation across model scales and merge recipes. We address both verification questions below.

Reproducibility release and matched 3-seed statistical verification (W1 / Q1)

Yes. We apologize for the incomplete archival notes in Appendices C and D due to an editing oversight during final preparation, and provide the complete training/evaluation scripts, configurations, verbatim prompts, deterministic parsers, and per-run logs in our supplementary archive.

  1. Exact checkpoints and training specifications. 2B merges OpenGVLab/InternVL3_5-2B-MPO and Qwen/Qwen3-1.7B (base Qwen/Qwen3-1.7B-Base); 8B merges lmms-lab/LLaVA-OneVision-1.5-8B-Instruct and Qwen/Qwen3-8B (base Qwen/Qwen3-8B-Base); 14B merges OpenGVLab/InternVL3_5-14B-MPO and Qwen/Qwen3-14B (base Qwen/Qwen3-14B-Base). Only homologous language-decoder layers (bfloat16) are merged, retaining the source MLLM's vision encoder, projector, embeddings, and LM head. Layer-wise fusion uses analytical weights [1]; TIES [2] uses top-20%20\% trimming and λ=1.0\lambda=1.0; DARE+TIES [3] uses drop rate p=0.9p=0.9 with 10×10\times rescaling (seed=0). Both stages use AdamW (lr=10−6\text{lr}=10^{-6}, β=(0.9,0.999)\beta=(0.9,0.999), weight decay 0.010.01, zero warmup, batch size 88, 1 epoch per stage, max lengths 8,192/32,7688{,}192 / 32{,}768, K=8K=8 rollouts at temperature 1.01.0, greedy anchor at temperature 0.00.0, ϵclip=0.2\epsilon_{\text{clip}}=0.2, βKL=0.1\beta_{\text{KL}}=0.1, dataloader seed=1, eval seed=42). All training queries and images were cross-checked against every benchmark split to confirm zero question or image overlap.
  2. Matched 3-seed comparison and Stage-II supervision asymmetry. Under the fixed-Stage-I protocol, all three RL methods start from the identical checkpoint θstab\theta_{\text{stab}} and vary only the Stage-II target atarget∈{amaj,a⋆,ar}a_{\text{target}} \in \{a_{\text{maj}}, a^\star, a_r\}. Note that Stage-II PASA uses zero ground-truth RL labels (Answer-Masked PASA in Table 3b also removes Stage-I final-answer loss and reaches 59.0 / 69.5 / 56.5 / 48.2 on MMMU / MathVista / MMStar / Hallusion, within 0.2 points of default PASA). Thus, Majority / MM-UPT [4] is the matched label-free RL baseline, whereas Gold-answer GRPO uses privileged reference answers. Matching ground-truth RL without RL labels would already be a strong result; across 3 independent RL seeds (mean ± SD, n=3n=3; two-sided Welch tt-statistics from unrounded seed scores, all p<0.02p < 0.02), PASA outperforms both Majority (+0.9 to +3.0) and Gold-answer GRPO (+0.5 to +2.6) on every benchmark. In the revision, we also streamline Section 3's role-update summary and standardize statistical notations across all captions:
Method (Fixed Stage-I Init)RL LabelsMMMUMathVistaMMStarOCRBenchHallusion
Stage I onlyNo57.465.353.780.546.7
+ Majority / MM-UPT [4]No57.8±0.2066.4±0.1954.0±0.1082.3±0.2146.4±0.16
+ Gold-answer GRPOYes58.7±0.1366.8±0.1754.3±0.1482.0±0.0547.1±0.11
+ Full PASA (Ours)No59.2±0.0669.4±0.2156.4±0.2583.2±0.0948.1±0.15
Δ\Delta vs. Majority (tt-stat)No+1.4 (11.6)+3.0 (18.4)+2.4 (15.4)+0.9 (6.8)+1.7 (13.4)
Δ\Delta vs. Gold GRPO (tt-stat)No+0.5 (6.1)+2.6 (16.7)+2.1 (12.7)+1.2 (20.2)+1.0 (9.3)

Why does Reasoning-Anchor GRPO beat Gold-Answer GRPO? (Q2)

Reasoning-Anchor GRPO produces a higher fraction of informative, nonzero agreement-variance rollout groups than Gold-Answer GRPO under finite K=8K=8 sampling while maintaining high target accuracy and co-evolving across training:

Rollout Metric (K=8K=8)No Stage I + ara_rStage I + MajorityStage I + GoldFull PASA
Format valid23.8%98.4%98.0%98.6%
Target hit (≥1/8\ge 1/8)66.7%74.0%69.0%77.0%
Agreement variance >0>052.0%67.0%62.0%71.0%
Initial role acc.73.9%47.2%100.0%75.3%
  1. Higher active gradient density than Gold GRPO. Whenever 0 of the K=8K=8 sampled perception trajectories matches atargeta_{\text{target}}, within-group agreement variance collapses to zero. On hard visual-math queries where post-merge perception is degraded, external answers a⋆a^\star go unmatched across all 8 candidates more often (69.0% hit; 62.0% nonzero agreement-variance groups), whereas same-model ara_r lies within the shared policy's reachable distribution (77.0% hit; 71.0% nonzero agreement-variance groups, +9.0 points over Gold GRPO).
  2. Stronger target role than Majority, plus shared-backbone co-evolution. Majority (a_maj, requiring a strict non-tie plurality among valid extracted answers, so tied groups yield no majority hit) aggregates the degraded perception role (47.2% greedy perception accuracy vs. 75.3% for ara_r). Moreover, as perception updates refine the shared parameters, ara_r itself improves from 75.3% to 83.4% on the 6,000 pool (78.0% →\to 80.5% →\to 82.7% →\to 86.6% on the 1,000-query monitor in Table 7).

Judge-free verification of visual extraction and blank-image controls (W2)

Yes, PASA genuinely recovers visual perception rather than merely aligning to speaker priors. While Stage-II's reward (0.9 agree+0.1 rfmt0.9\,\text{agree}+0.1\,r_{\text{fmt}}) optimizes answer agreement and format and VGS (Table 6) is a supplementary trace audit, all three of our perceptual benchmarks require zero LLM judging and, paired with a blank-image control, directly verify visual extraction:

Perceptual BenchmarkSourceDirect MergeStage IMajorityGold GRPOFull PASA
OCRBench (0 LLM judge)82.579.480.582.3±0.2182.0±0.0583.2±0.09
MMStar (rule-graded MCQ)55.651.253.754.0±0.1054.3±0.1456.4±0.25
HallusionBench (rule-graded Y/N)47.545.546.746.4±0.1647.1±0.1148.1±0.15
  1. Judge-free character, document, and formula reading (OCRBench). OCRBench uses deterministic string/exact matching with zero LLM involvement. Agreeing with a reasoning role's answer style cannot read unseen scene text, digits, or formulas from an image, yet PASA recovers OCRBench above the unmerged source across all three scales (82.5 →\to 79.4 →\to 83.2 at 2B, beating Gold 82.0 and Majority 82.3; 82.9 →\to 76.2 →\to 83.3 at 8B; 82.9 →\to 78.9 →\to 83.7 at 14B).
  2. Blank-image visual-dependence test. Replacing input images with a blank white canvas causes single-run PASA accuracy to drop by 59.9 points on MathVista (69.5% →\to 9.6%) and 50.9 points on MMStar (56.6% →\to 5.7%) (46% of MathVista items are open-ended numerical questions, and on MMStar an empty canvas frequently leads Pperc\mathcal{P}_{\text{perc}} to report missing/unanswerable visual content rather than guess blind option letters), confirming direct dependence on visual input.
  3. Wrong-anchor override without length inflation. Across all wrong-anchor items (ar≠a⋆a_r \neq a^\star, 16.6% of the 6,000 pool at 2B), the complete 4-way partition is 43.2% perception correct (ap=a⋆a_p=a^\star), 21.8% follows wrong anchor (ap=ara_p=a_r), 30.9% other wrong, and 4.1% parse failure (66.5% override vs. 33.5% follow on ap∈{a⋆,ar}a_p \in \{a^\star,a_r\} on this 6,000-query pool; Table 4's 68% / 32% at 2B and 87% / 13% at 8B were evaluated on a separate benchmark diagnostic cohort), at unchanged output length (1,512 vs. 1,528 tokens).

[1] Wei & Chen, Training-Free Reasoning and Reflection in MLLMs, 2025.

[2] Yadav et al., TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023.

[3] Yu et al., Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch, ICML 2024.

[4] Wei et al., Unsupervised Post-Training for Multi-Modal LLM Reasoning via GRPO, 2025.

Dear Reviewer,

Thank you again for your thoughtful review and for the time you have spent evaluating our paper.

We have carefully addressed the concerns raised in your review and hope that our rebuttal has helped clarify the issues you pointed out.

As the discussion period is coming to an end, we would like to kindly ask if you have any remaining questions or if there is anything further we can clarify. We would be more than happy to provide additional explanations.

Thank you very much for your time and consideration.

Reviewer_92DAOpenReview ↗
Rating
1(−1)
Confidence
4(+2)
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper studies a trade-off observed when merging reasoning-specialized language models into multimodal models: reasoning-oriented benchmark scores improve, while some visual perception scores decline. The authors propose PASA, a two-stage post-merge training procedure. The first stage uses supervised fine-tuning on synthetic traces to teach structured visual-evidence generation. The second stage uses two prompted roles of the same model: a reasoning role produces answer targets, while a perception role generates evidence traces and answers. Perception trajectories are optimized with GRPO using answer agreement and format rewards. Experiments across three model sizes and several merging recipes report improvements in both perception and reasoning benchmarks.

Critical strengths
  1. The paper addresses a useful problem. Transferring reasoning capabilities through model merging is less useful if it damages visual grounding. Examining both gains and losses is a sensible way to evaluate the resulting models.
  2. The reported improvements cover several settings. PASA improves all five main benchmark scores over the corresponding merged checkpoints across the evaluated model sizes and merge recipes. The stage-wise results also suggest that both supervised stabilization and subsequent RL contribute to the gains.
  3. Several controls help clarify what the method uses. The answer-masked SFT control and the self-generated trace variant address dependence on direct answer supervision and an external trace generator. The separate visual-evidence audit is also useful, since the training reward does not directly check visual correctness.
Critical weaknesses
  1. The central comparisons do not clearly isolate the benefit of cross-role supervision. Table 2 presents the fixed-Stage-I comparisons as evidence for using reasoning-role answers rather than majority targets or gold answers. However, Appendix C.3 states that the retained records do not fully specify whether candidate prompts, sampling, reward components, clipping, KL, and other settings were matched across these runs. These choices can substantially affect RL outcomes. Sharing the initialization and training queries is therefore insufficient to attribute the improvement to the proposed supervision direction. The fact that PASA outperforms gold-answer GRPO makes this ambiguity particularly relevant: it is difficult to understand what drives the ordering without knowing which other settings differed. I appreciate the explicit qualification in the appendix, but it leaves the main methodological claim insufficiently supported.
  2. The connection between post-merge asymmetry and the proposed learning signal remains incomplete. The paper observes two different gaps: a change in benchmark performance after merging, and a difference in answer accuracy between two prompts after Stage I. The second gap motivates the actual training procedure, but it does not by itself connect that procedure to the first observation. As the authors acknowledge, the role gap has not been shown to arise from merging. The results support PASA as a potentially useful post-training procedure, but provide less evidence that it specifically exploits retained reasoning capability to repair merge-induced perception damage. This also makes the conceptual contribution beyond structured SFT followed by cross-prompt self-training less clear.
  3. Important experimental details are unavailable, limiting reproducibility and confidence in the supporting analyses. Appendices C and D explicitly note missing checkpoint identifiers, merge coefficients and parameter-group details, resolved run configurations, and exact training prompts. These are central ingredients of the proposed pipeline. Some diagnostic results have similar limitations: Appendix F.2 lacks the item-level accounting needed to reconcile wrong-anchor outcomes, while Appendix F.3 cannot distinguish nonzero rewards from nonzero within-group reward variance in the reported aggregate statistics. The qualifications are transparent, but the missing information substantially limits what can be concluded from these analyses.
Questions for authors

For the fixed-Stage-I comparisons in Table 2, were the perception prompts, rollout settings, reward components, and optimization settings actually identical across PASA, gold-answer GRPO, and MM-UPT? Appendix C.3 leaves this unclear, and my main concern is whether the gains can be attributed to the reasoning-role targets or to other differences between the training procedures.

4 comments

We thank Reviewer 92DA for the careful review and for noting the practical value of repairing post-merge visual grounding loss, our consistent gains across scales and merge recipes, and our answer-masked, self-generated, and answer-blind grounding controls. We address all three concerns below.

Were the fixed-Stage-I comparisons strictly matched, and why does PASA outperform Gold-Answer GRPO? (W1 / Q1)

Yes. Across Stage I + Majority (MM-UPT) [1], Stage I + Gold-answer GRPO, and Full PASA, every non-target setting is strictly identical. All three runs start from the same Stage-I checkpoint θstab\theta_{\text{stab}}, train for 1 epoch on the same 6,000 MMPR-Tiny queries under the same structured perception prompt PpercP_{\text{perc}}, sample K=8K=8 rollouts at temperature 1.01.0, use the identical reward 0.9 agree+0.1 rfmt0.9\,\text{agree}+0.1\,r_{\text{fmt}}, and apply the same AdamW (lr=10−6\text{lr}=10^{-6}, batch 88), GRPO clipping (ϵclip=0.2\epsilon_{\text{clip}}=0.2), and KL settings (βKL=0.1\beta_{\text{KL}}=0.1 relative to fixed θstab\theta_{\text{stab}}). Only the agreement target atarget∈{amaj,a⋆,ar}a_{\text{target}} \in \{a_{\text{maj}}, a^\star, a_r\} changes. We apologize for the incomplete configuration note in Appendix C.3 due to an editing oversight during final preparation, and report the full 3-seed comparison below:

Method (Fixed Stage-I Init)RL LabelsMMMUMathVistaMMStarOCRBenchHallusion
Stage I onlyNo57.465.353.780.546.7
+ Majority / MM-UPT [1]No57.8±0.2066.4±0.1954.0±0.1082.3±0.2146.4±0.16
+ Gold-answer GRPOYes58.7±0.1366.8±0.1754.3±0.1482.0±0.0547.1±0.11
+ Full PASA (Ours)No59.2±0.0669.4±0.2156.4±0.2583.2±0.0948.1±0.15
Δ\Delta vs. MajorityNo+1.4+3.0+2.4+0.9+1.7
Δ\Delta vs. Gold GRPONo+0.5+2.6+2.1+1.2+1.0

Table 2a's single-run scores (57.8 / 66.7 / 53.9 / 82.1 / 46.5 for Majority, 58.8 / 66.7 / 54.3 / 81.9 / 46.9 for Gold GRPO, and 59.2 / 69.5 / 56.6 / 83.2 / 48.1 for PASA; Table 9 previously omitted PASA's OCRBench 3-run summary; entries show mean ± sample SD across n=3n=3 RL seeds) align closely with these 3-seed means. Notably, Stage-II PASA uses zero ground-truth RL labels (Majority is the matched label-free RL baseline, whereas Gold-answer GRPO uses privileged reference answers). Matching ground-truth RL without RL labels would already validate cross-role alignment; PASA surpasses both Majority (+0.9 to +3.0) and Gold-answer GRPO (+0.5 to +2.6) on every benchmark.

Rollout diagnostics over the 6,000 RL queries explain why ara_r outperforms a⋆a^\star under a finite 1-epoch budget:

Rollout Metric (K=8K=8)No Stage I + ara_rStage I + MajorityStage I + GoldFull PASA
Format valid23.8%98.4%98.0%98.6%
Target hit ($ ≧ 1/8$)66.7%74.0%69.0%77.0%
Agreement variance >0>052.0%67.0%62.0%71.0%
  1. Denser informative GRPO updates than Gold GRPO. When none of the K=8K=8 perception rollouts matches atargeta_{\text{target}}, within-group agreement variance is zero. On hard visual-reasoning queries where post-merge perception is degraded, external answers a⋆a^\star go unmatched across all 8 rollouts more often (69.0% hit; 62.0% nonzero agreement-variance groups), whereas the same-model anchor ara_r lies closer to the shared policy's reachable distribution (77.0% hit; 71.0% nonzero agreement-variance groups, +9.0 points over Gold GRPO).
  2. Stronger target role than Majority, plus shared-backbone co-evolution. Majority (a_maj, requiring a strict non-tie plurality among valid extracted answers, so tied groups yield no majority hit) aggregates the degraded perception role (47.2% greedy perception accuracy vs. 75.3% for ara_r). Because both roles share parameters, perception updates also improve ara_r across training (75.3% →\to 83.4% on the 6,000 pool; 78.0% →\to 80.5% →\to 82.7% →\to 86.6% on the fixed 1,000 subset in Table 7).

Does the role-reliability gap come from model merging rather than prompting or Stage-I SFT? (W2)

Yes. Evaluating both roles (PreasonP_{\text{reason}} and PpercP_{\text{perc}}) on the exact same 6,000 queries across {Unmerged Source, Merged} × {Before, After Stage-I SFT} shows that cross-modal merging directly creates and reverses the role gap prior to any SFT:

Checkpoint (2B, 6k Pool)MergedStage IReasoning Acc.Perception Acc.Gap (R−P\text{R}-\text{P})
Source MLLMNoNo54.7%69.6%−14.9
Direct MergeYesNo73.9%42.5%+31.4
Source + Stage INoYes57.0%71.2%−14.2
Merge + Stage I (θstab\theta_{\text{stab}})YesYes75.3%47.2%+28.1
Full PASAYesYes83.4%77.5%+5.9

On the unmerged Source MLLM, perception exceeds reasoning by 14.9 points (69.6% vs. 54.7%), and applying the identical 20,000-trace Stage-I SFT (Source + Stage I) leaves this negative gap virtually unchanged at −14.2 (71.2% vs. 57.0%). Directly merging the models (Direct Merge) immediately boosts reasoning (+19.2) while degrading perception (−27.1), reversing the role gap by +46.3 points (−14.9 →\to +31.4 before SFT; +28.1 after SFT). Thus, PASA's conceptual contribution goes beyond generic cross-prompt self-training: it diagnoses the merge-induced reliability inversion between reasoning and perception and directs detached, parameter-shared supervision specifically from the retained reasoning role to repair the degraded perception role (47.2% →\to 77.5%).

Complete model, hyperparameter, prompt, and diagnostic specifications (W3: Appendices C, D, F)

We apologize for the omitted archival details in Appendices C, D, and F, and provide every exact specification below (in the revision, we also streamline Section 3 and standardize all table captions and statistical conventions for clarity):

  1. Checkpoints and merge parameters (Appendices C.1 & C.4). 2B merges OpenGVLab/InternVL3_5-2B-MPO and Qwen/Qwen3-1.7B (base Qwen/Qwen3-1.7B-Base); 8B merges lmms-lab/LLaVA-OneVision-1.5-8B-Instruct and Qwen/Qwen3-8B (base Qwen/Qwen3-8B-Base); 14B merges OpenGVLab/InternVL3_5-14B-MPO and Qwen/Qwen3-14B (base Qwen/Qwen3-14B-Base). Only homologous language-decoder layers (bfloat16) are merged, preserving the MLLM's vision encoder, projector, embeddings, and LM head. Layer-wise fusion uses analytical task-vector weights [2]; TIES [3] uses top-20%20\% trimming and λ=1.0\lambda=1.0; DARE+TIES [4] uses drop rate p=0.9p=0.9 with 10×10\times rescaling (seed=0).
  2. Training configuration, prompts, and grading (Appendices C.2 & D). Both stages run for 1 epoch with AdamW (lr=10−6\text{lr}=10^{-6}, β=(0.9,0.999)\beta=(0.9,0.999), weight decay 0.010.01, zero warmup, global batch size 88) on 4×A100 GPUs. Stage I trains on 20,000 Qwen3-VL-Max structured traces (max length 8,1928{,}192) filtered for all six caption sections (Image category, Scene summary, Explicit components, Relationships, Inferred properties, Ambiguities); Stage II trains on 6,000 MMPR-Tiny queries (max prompt/response 8,192/32,7688{,}192/32{,}768, K=8K=8 rollouts at temperature 1.01.0, greedy anchor at temperature 0.00.0, ϵclip=0.2\epsilon_{\text{clip}}=0.2, βKL=0.1\beta_{\text{KL}}=0.1), enforcing outer <caption>...</caption> and \boxed{} structure and symbolic math equivalence grading. All training queries and images were cross-checked against every evaluation split to confirm zero question or image overlap.
  3. Complete wrong-anchor and group-variance accounting (Appendices F.2 & F.3). Across all wrong-anchor items (ar≠a⋆a_r \neq a^\star, 16.6% of the 6,000 pool at 2B), the complete 4-way outcome breakdown summing to 100% is: 43.2% perception correct (ap=a⋆a_p=a^\star), 21.8% follows wrong anchor (ap=ara_p=a_r), 30.9% other wrong answer, and 4.1% parse failure (66.5% vs. 33.5% on the decisive subset ap∈{a⋆,ar}a_p \in \{a^\star,a_r\} on this 6,000-query pool; Table 4's 68% / 32% at 2B and 87% / 13% at 8B were evaluated on a separate benchmark diagnostic cohort). In Appendix F.3, 98.6% format validity, 77.0% group target reachability, and 71.0% nonzero agreement-variance groups (77.6% nonzero total-reward variance including format differences) clarify the group-level update statistics.

[1] Wei et al., Unsupervised Post-Training for Multi-Modal LLM Reasoning via GRPO, 2025.

[2] Wei & Chen, Training-Free Reasoning and Reflection in MLLMs, 2025.

[3] Yadav et al., TIES-Merging: Resolving Interference When Merging Models, NeurIPS 2023.

[4] Yu et al., Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch, ICML 2024.

Dear Reviewer,

Thank you again for your thoughtful review and for the time you have spent evaluating our paper.

We have carefully addressed the concerns raised in your review and hope that our rebuttal has helped clarify the issues you pointed out.

As the discussion period is coming to an end, we would like to kindly ask if you have any remaining questions or if there is anything further we can clarify. We would be more than happy to provide additional explanations.

Thank you very much for your time and consideration.

Dear Reviewer 92DA,

As the discussion period draws to a close, we wanted to kindly check whether our response and revised manuscript—including our matched 3-seed Stage-II target comparisons and active-RL-group rollout diagnostics (+9.0 pp informative groups over Gold GRPO), our 6,000-query isolation of the merge-induced role-reliability inversion, and the full checkpoint, prompt, and hyperparameter specifications in Appendices C, D, and F—have addressed your questions, and we would be happy to clarify any remaining points.

Thank you very much for your time and consideration.

Comments on the paper

2 comments

We thank the Area Chair and Reviewers (92DA, tFCR, 72Gb) for their constructive feedback and for highlighting the practical value of repairing post-merge visual grounding loss, PASA's parameter-shared cross-role design, and our consistent gains across model scales (2B, 8B, 14B) and merge recipes (layer-wise, TIES, DARE+TIES). Below we directly answer the four core questions shared across your reviews.

1. Does the reasoning-over-perception role gap arise from model merging rather than prompting or Stage-I SFT?

Yes. Evaluating both prompted roles (PreasonP_{\text{reason}} and PpercP_{\text{perc}}) on the exact same 6,000 queries across {Unmerged Source, Merged} × {Before, After Stage-I SFT} isolates the effect of merging directly:

Checkpoint (2B, 6k Pool)MergedStage IReasoning Acc.Perception Acc.Gap (R−P\text{R}-\text{P})
Source MLLMNoNo54.7%69.6%−14.9
Direct MergeYesNo73.9%42.5%+31.4
Source + Stage INoYes57.0%71.2%−14.2
Merge + Stage I (θstab\theta_{\text{stab}})YesYes75.3%47.2%+28.1
Full PASAYesYes83.4%77.5%+5.9

On the unmerged Source MLLM, perception exceeds reasoning (69.6% vs. 54.7%, gap −14.9), and applying the identical 20k structured Stage-I SFT (Source + Stage I) leaves this negative gap virtually unchanged at −14.2 (71.2% vs. 57.0%). Cross-modal merging (Direct Merge) immediately boosts reasoning (+19.2) while degrading perception (−27.1), reversing the role gap by +46.3 points (−14.9 →\to +31.4) prior to any SFT. PASA then leverages this retained reasoning strength to restore perception (47.2% →\to 77.5%).

2. Under strictly matched fixed-Stage-I RL settings and 3 seeds, how reliable are PASA's gains?

Holding the Stage-I checkpoint θstab\theta_{\text{stab}}, 6,000 MMPR-Tiny queries, structured perception prompt PpercP_{\text{perc}}, K=8K=8 sampling, 0.9 agree+0.1 rfmt0.9\,\text{agree}+0.1\,r_{\text{fmt}} reward, and AdamW/GRPO/KL hyperparameters strictly identical while varying only atarget∈{amaj,a⋆,ar}a_{\text{target}} \in \{a_{\text{maj}}, a^\star, a_r\}, 3 independent RL seeds confirm consistent, non-overlapping gains across all 10 comparisons (p < 0.02 on every Welch two-sample test):

Method (Fixed Stage-I Init)RL LabelsMMMUMathVistaMMStarOCRBenchHallusion
Stage I onlyNo57.465.353.780.546.7
+ Majority / MM-UPT (Wei et al., 2025)No57.8±0.2066.4±0.1954.0±0.1082.3±0.2146.4±0.16
+ Gold-answer GRPOYes58.7±0.1366.8±0.1754.3±0.1482.0±0.0547.1±0.11
+ Full PASA (Ours)No59.2±0.0669.4±0.2156.4±0.2583.2±0.0948.1±0.15
Δ\Delta vs. MajorityNo+1.4+3.0+2.4+0.9+1.7
Δ\Delta vs. Gold GRPONo+0.5+2.6+2.1+1.2+1.0

Note the supervision asymmetry: Stage-II PASA uses zero ground-truth RL labels (Answer-Masked PASA in Table 3b also zeroes Stage-I final-answer loss and reaches 59.0 / 69.5 / 56.5 / 48.2 on MMMU / MathVista / MMStar / Hallusion). Majority / MM-UPT is thus the matched label-free RL baseline, whereas Gold-answer GRPO uses privileged reference answers. Matching ground-truth RL without RL labels would already validate cross-role alignment; PASA surpasses both Majority (+0.9 to +3.0) and Gold-answer GRPO (+0.5 to +2.6) on every benchmark.

3. Why does label-free Reasoning-Anchor GRPO outperform Gold-Answer GRPO, and how does Stage I interact with Stage II?

Completing the 2×3 {Without Stage I, With Stage I} × {Majority, Gold, Reasoning} comparison (No Stage I + Reasoning scores 57.7 / 65.1 / 53.4 / 80.9 / 47.0, giving a positive relative-to-Gold interaction IR−G=(YesR−YesG)−(NoR−NoG)I_{R-G} = (\text{Yes}_R-\text{Yes}_G)-(\text{No}_R-\text{No}_G) of +0.9 / +3.2 / +2.3 / +0.6 / +0.6 on single runs and +1.0 / +3.0 / +2.1 / +0.5 / +0.4 using With-Stage-I 3-seed means) alongside 6k rollout diagnostics explains both effects:

Rollout Metric (K=8K=8)No Stage I + ara_rStage I + MajorityStage I + GoldFull PASA
Format valid23.8%98.4%98.0%98.6%
Target hit (≥1/8\ge 1/8)66.7%74.0%69.0%77.0%
Agreement variance >0>052.0%67.0%62.0%71.0%

Without Stage I, Direct Merge satisfies the structured <caption>...</caption> + \boxed{} format on only 23.8% of rollouts (52.0% nonzero agreement-variance groups). With Stage I (98.0%–98.6% valid format), external gold answers a⋆a^\star on hard visual-math items still miss all K=8K=8 perception rollouts more often (69.0% hit; 62.0% nonzero agreement-variance groups), whereas same-model ara_r is both grounded in a much stronger role than Majority (75.3% reasoning vs. 47.2% greedy perception) and distributionally reachable by the shared policy (77.0% hit; 71.0% nonzero agreement-variance groups, +9.0 points over Gold GRPO) while co-evolving upward (75.3% →\to 83.4%).

4. Does PASA genuinely recover visual extraction without relying on an LLM judge, and how will presentation be improved?

  • Judge-free visual verification: Yes. An LLM judge is used only for the supplementary VGS trace audit in Table 6 (64.7 →\to 68.2 →\to 75.9), whereas all three perceptual benchmarks confirm genuine visual recovery with zero LLM judging. OCRBench uses deterministic string/exact matching with zero LLM involvement and recovers above the unmerged source across all scales (82.5 →\to 79.4 →\to 83.2 at 2B; 82.9 →\to 76.2 →\to 83.3 at 8B; 82.9 →\to 78.9 →\to 83.7 at 14B), alongside rule-graded MMStar (51.2 →\to 56.4±0.25) and HallusionBench (45.5 →\to 48.1±0.15). Replacing input images with a blank white canvas causes PASA's accuracy to drop from 69.5% →\to 9.6% on MathVista and 56.6% →\to 5.7% on MMStar, and across all wrong-anchor items (ar≠a⋆a_r \neq a^\star) on the 6k pool, the complete 4-way outcome breakdown summing to 100% is 43.2% perception correct, 21.8% follows wrong anchor, 30.9% other wrong, and 4.1% parse failure.
  • Clarity and presentation: In the revision, we streamline Section 3 with a concise step-by-step role-update summary, place complete hyperparameter, prompt, and denominator tables in Appendices C–F, and state exact sample counts and statistical conventions (mean ± SD, n=3n=3) in every table caption.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.