- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents MERIT, a three-stage framework for multimodal table reasoning. The method first adapts the visual encoder using table reconstruction, localization, and color-conditioned objectives, then performs multimodal SFT on verified evidence-grounded reasoning trajectories, and finally applies GRPO with decomposed rewards for evidence selection, calculation, final-answer correctness, and output format. Experiments cover table QA and fact verification, including both in-domain and additional out-of-domain benchmarks.
Critical strengths
- The evidence-grounded reasoning formulation provides explicit and verifiable intermediate supervision.
- The paper includes ablations of the three training stages, visual adaptation objectives, and individual reward components, as well as evaluations on additional datasets and an error analysis.
Critical weaknesses
1. The methodological novelty over closely related table-specific alignment and RL approaches is not sufficiently established. The stage 1 in MERIT is closely related to DiSCo-style table-specific visual alignment [1], while its SFT-to-GRPO pipeline with intermediate rewards is conceptually close to Table-R1 [2]. Although MERIT uses different supervision and reward decomposition, the paper does not clearly establish a sufficiently distinct methodological contribution, and it also lacks direct comparisons with these two closely related methods under comparable settings.
2. The ablations suggest that most of the performance gain comes from grounded SFT, while the proposed specialized components provide only modest additional improvements. SFT alone improves the average score from 76.05 to 84.46, whereas the full pipeline reaches 85.45. Moreover, the decomposed reward improves over answer-only GRPO by only 0.44. These results make it unclear how much of MERIT’s overall gain is attributable to the proposed encoder adaptation and fine-grained reward design.
[1] Yingjie Zhu, Xuefeng Bai, Kehai Chen, Yang Xiang, Youcheng Pan, Xiaoqiang Zhou, and Min Zhang. Decoupling skeleton and flesh: Efficient multimodal table reasoning with disentangled alignment and structure-aware guidance. ICML 2026.
[2] Xiaoqiang Kang, Shengen Wu, Zimu Wang, Yilin Liu, Xiaobo Jin, Kaizhu Huang, Wei Wang, Yutao Yue, Xiaowei Huang, and Qiufeng Wang. Can GRPO boost complex multimodal table understanding? EMNLP 2025.
Questions for authors
- Why does encoder adaptation + SFT underperform SFT without encoder adaptation (83.34 vs. 84.46), while the benefit of encoder adaptation only emerges after GRPO?
- For the OOD datasets without gold serialized tables, the appendix states that MinerU-generated Markdown is provided to visual models together with the table image. Can the authors also report a strictly image-only OOD evaluation to support the claimed generalization when structured representations are unavailable?
6 comments
Response to Reviewer XZ19 (Part 2/3)
W2. Gains mainly from grounded SFT (continued)
Moreover, each stage makes subsequent training more effective. To assess rollout quality before GRPO, we sample eight responses per prompt on 512 RL prompts using the base model and the two SFT checkpoints from Table 4 (Appendix C.6, Table 14):
| Starting point | Format error (%) | Zero-variance groups (%) | Prompts with no correct rollout (%) | pass@8 | GRPO gain |
|---|---|---|---|---|---|
| Base | 100.00 | 57.23 | 27.15 | 72.85 | – |
| SFT, original encoder | 2.32 | 1.95 | 17.19 | 82.81 | +0.04 |
| SFT, adapted encoder | 0.81 | 1.95 | 15.82 | 84.18 | +2.11 |
Before SFT, no response contains all required fields, and 57.23% of GRPO groups receive identical rewards across all 8 rollouts, so GRPO has almost no gradient to learn from. SFT reduces this share to 1.95%, which is what makes GRPO trainable.
Contribution of encoder adaptation and RL. In Table 4, GRPO improves accuracy by only 0.04 points with the original encoder, compared with 2.11 points with the adapted encoder. This indicates that the adapted encoder unlocks the potential of RL, and also confirms the contribution of GRPO. During RL, a higher pass@k provides more opportunities to sample correct trajectories and obtain answer-level reinforcement signals (Yue et al., 2025). With the original encoder, 17.19% of prompts have no correct rollout and therefore receive no positive answer-correctness reward. The adapted encoder reduces this share to 15.82% and raises pass@8 from 82.81 to 84.18, despite having lower SFT accuracy (83.34 vs. 84.46). Thus, adaptation improves the availability of correct trajectories for subsequent GRPO, even without improving the initial SFT accuracy.
This interaction persists when both models start GRPO from comparable SFT accuracies.
- Setting of the 120-step GRPO runs. To compare GRPO from matched starting points, we use two SFT checkpoints with comparable accuracy: the adapted model trained for one additional SFT epoch (84.39) and the non-adapted SFT model from Table 4 (84.46). From each, we run 120 GRPO steps with the same RL data, hyperparameters, and full reward as in the paper. Due to limited computational resources, this is shorter than the 780 steps used for the final model, so the absolute gains are smaller than those in Table 4.
- Result. Under this setting, the adapted model gains 0.53 points (95% CI [+0.10, +0.84]), compared with 0.09 points without adaptation.
Beyond RL, adaptation improves table perception from 46.89 to 53.51 in Table 1, and the complete pipeline outperforms the non-adapted one by 0.95 points on average, with gains on each of the five datasets in Table 4.
Decomposed reward. Under the same 120-step setting, removing the evidence or calculation reward reduces the 0.53-point GRPO gain to 0.32 or 0.31 points, which are no longer significant. The process rewards also reduce the share of GRPO groups without gradient from 49.2% under an answer-only reward to 1.95%. Our response to Reviewer 5QpC gives the full ablation (Appendix C.7).
Response to Reviewer XZ19 (Part 3/3)
Q1. Encoder adaptation + SFT versus SFT
- "Why does encoder adaptation + SFT underperform SFT without encoder adaptation (83.34 vs. 84.46), while the benefit of encoder adaptation only emerges after GRPO?"
(1) Stage 1 changes the visual features, so the decoder must realign to them during SFT, and the SFT budget in Table 4 is not sufficient. With one additional SFT epoch, the gap closes (84.39 vs. 84.46; Appendix C.6, Table 13). The benefit then emerges after GRPO because adaptation raises the pass@8 ceiling that bounds GRPO, as explained in W2.
Q2. Strict image-only OOD evaluation
- "Can the authors also report a strictly image-only OOD evaluation to support the claimed generalization when structured representations are unavailable?"
(1) We removed all textual tables, namely the MinerU Markdown for FinMMR, TableVista, and MMMU-T and the serialized tables for HiTab and FinQA (Appendix C.8, Table 18):
| Model (image only) | HiTab | FinQA | FinMMR | TableVista | MMMU-T | Avg. |
|---|---|---|---|---|---|---|
| Qwen3-VL-8B | 76.90 | 26.24 | 30.52 | 65.67 | 43.86 | 48.64 |
| MERIT | 77.98 | 28.25 | 44.62 | 63.78 | 40.35 | 51.00 |
Without any structured text, MERIT still outperforms the backbone by 2.36 points on average, which is significant with a 95% CI of [+0.70, +3.99], and the FinMMR gain of 14.10 points persists. The MMMU-T gap comes from missing subject knowledge: under the Table 3 setting, adding Wikipedia retrieval closes the gap completely, with both models reaching 45.03, as shown in our response to Reviewer 5QpC (W2 & Q5).
We hope these responses clarify our contributions, and we would be grateful if the reviewer would consider raising the score.
References
- Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025. arXiv:2504.13837.
- Lai Wei, Yuting Li, Kaipeng Zheng, Chen Wang, Yue Wang, Linghe Kong, Lichao Sun, and Weiran Huang. Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start. arXiv:2505.22334, 2025.
- Marah Abdin et al. Phi-4-reasoning Technical Report. arXiv:2504.21318, 2025.
Response to Reviewer XZ19 (Part 1/3)
We thank the reviewer for the constructive feedback. We address each concern below; the new experimental results have been added to the revised PDF (Appendices C.6 and C.8).
W1. Novelty relative to DiSCo and Table-R1
(1) MERIT differs from both methods in what it trains and how it supervises reasoning.
The contribution of MERIT is to connect table-specific encoder adaptation with evidence-grounded reasoning and executable verification of intermediate steps in one framework.
(2) Neither method has released code or weights, so a controlled comparison is not possible. Their reported numbers also use different evaluation protocols: Qwen3-VL-8B scores 36.01 on TAT-QA in the DiSCo paper but 82.51 under ours. We therefore compare each method's gain over its own backbone, with the backbone score in parentheses:
* Table-R1 reports 7.40 for its backbone on TabFact.
MERIT achieves the largest reported gains on WTQ and TAT-QA, despite starting from stronger baselines. On TabFact, Table-R1 reports a larger gain (+66.00), but from a much lower baseline (7.40). On InfoTabs and HiTab, the other methods start from baselines 27 to 44 points lower and thus have far more headroom. Moreover, HiTab is part of Table-R1's training data but unseen by MERIT.
W2. Gains mainly from grounded SFT
We first summarize our reading of Table 4 and then give the details.
Improvements of SFT. Grounded SFT is not a generic baseline to which the other components are added; it is itself one of MERIT's components. Its training data are our evidence-grounded trajectories, which cite every cell by coordinate and value and are verified for evidence, derivation, and answer. Encoder adaptation, grounded SFT, and decomposed-reward GRPO are designed as one method, so the gain from grounded SFT is part of MERIT's gain. With all three stages, MERIT improves over the backbone by 9.40 points, up from 7.29 points after adapted SFT. That SFT accounts for the largest share is also typical of SFT-then-RL pipelines: for multimodal reasoning, cold-start SFT raises Qwen2.5-VL-3B from 40.00 to 49.46 and GRPO adds only 1.38 points (Wei et al., 2025); Phi-4-reasoning gains 44.7 points on OmniMath from SFT and 5.3 points from RL (Abdin et al., 2025). The RL share in MERIT is larger than in these pipelines.