acceptodds
Under review as a conference paper at ICLR 2027

MERIT: Multimodal Evidence Reasoning over Image Tables

Abstract

Table reasoning with language models typically assumes access to clean textual tables. In practice, the textual tables are inputted as sequence, which can corrupt cell contents, distort structural relationships, and discard visual cues, leaving the model to reason over incomplete evidence. In contrast, table images retain this information, but using it requires both accurate perception and reasoning grounded in the relevant cells. Thereafter, we propose MERIT (Multimodal Evidence Reasoning over Image Tables), a three-stage framework for learning to reason over table images, with serialized tables serving as optional auxiliary inputs (if provided). Our analysis identifies visual evidence acquisition as a bottleneck, motivating table-specific encoder adaptation through reconstruction, localization, and color-conditioned supervision. Building on the adapted representations, supervised fine-tuning teaches explicit evidence selection and reasoning using verified trajectories. Finally. reinforcement learning refines the policy with separate rewards for evidence correctness, calculation, final answers, and output format. By providing feedback on intermediate steps, the framework targets retrieval and computational errors that final-answer supervision alone does not distinguish. We evaluate the approach on table question answering and fact verification, analyze the contributions of visual inputs and training components, and assess generalization to additional datasets with different table structures and visual presentations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Reasoning, Table Question Answering, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XZ19OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MERIT, a three-stage framework for multimodal table reasoning. The method first adapts the visual encoder using table reconstruction, localization, and color-conditioned objectives, then performs multimodal SFT on verified evidence-grounded reasoning trajectories, and finally applies GRPO with decomposed rewards for evidence selection, calculation, final-answer correctness, and output format. Experiments cover table QA and fact verification, including both in-domain and additional out-of-domain benchmarks.

Critical strengths
  • The evidence-grounded reasoning formulation provides explicit and verifiable intermediate supervision.
  • The paper includes ablations of the three training stages, visual adaptation objectives, and individual reward components, as well as evaluations on additional datasets and an error analysis.
Critical weaknesses

1. The methodological novelty over closely related table-specific alignment and RL approaches is not sufficiently established. The stage 1 in MERIT is closely related to DiSCo-style table-specific visual alignment [1], while its SFT-to-GRPO pipeline with intermediate rewards is conceptually close to Table-R1 [2]. Although MERIT uses different supervision and reward decomposition, the paper does not clearly establish a sufficiently distinct methodological contribution, and it also lacks direct comparisons with these two closely related methods under comparable settings.

2. The ablations suggest that most of the performance gain comes from grounded SFT, while the proposed specialized components provide only modest additional improvements. SFT alone improves the average score from 76.05 to 84.46, whereas the full pipeline reaches 85.45. Moreover, the decomposed reward improves over answer-only GRPO by only 0.44. These results make it unclear how much of MERIT’s overall gain is attributable to the proposed encoder adaptation and fine-grained reward design.

[1] Yingjie Zhu, Xuefeng Bai, Kehai Chen, Yang Xiang, Youcheng Pan, Xiaoqiang Zhou, and Min Zhang. Decoupling skeleton and flesh: Efficient multimodal table reasoning with disentangled alignment and structure-aware guidance. ICML 2026.

[2] Xiaoqiang Kang, Shengen Wu, Zimu Wang, Yilin Liu, Xiaobo Jin, Kaizhu Huang, Wei Wang, Yutao Yue, Xiaowei Huang, and Qiufeng Wang. Can GRPO boost complex multimodal table understanding? EMNLP 2025.

Questions for authors
  1. Why does encoder adaptation + SFT underperform SFT without encoder adaptation (83.34 vs. 84.46), while the benefit of encoder adaptation only emerges after GRPO?
  2. For the OOD datasets without gold serialized tables, the appendix states that MinerU-generated Markdown is provided to visual models together with the table image. Can the authors also report a strictly image-only OOD evaluation to support the claimed generalization when structured representations are unavailable?
6 comments

Response to Reviewer XZ19 (Part 1/3)

We thank the reviewer for the constructive feedback. We address each concern below; the new experimental results have been added to the revised PDF (Appendices C.6 and C.8).

W1. Novelty relative to DiSCo and Table-R1

  1. "The methodological novelty over closely related table-specific alignment and RL approaches is not sufficiently established."
  2. "It also lacks direct comparisons with these two closely related methods under comparable settings."

(1) MERIT differs from both methods in what it trains and how it supervises reasoning.

  • Compared with DiSCo. DiSCo aligns an LVLM to table structure and content with LoRA, and its reasoning module, Table-GLS, is a training-free prompting pipeline. MERIT trains the visual encoder itself, including a Color objective for visual attributes, and learns reasoning through SFT and RL.
  • Compared with Table-R1. Table-R1 rewards table recognition with TEDS and rewards the final answer. MERIT rewards the reasoning trace itself with deterministic, decomposed rewards: a cited cell earns credit only if its row, column, and value all match, and each arithmetic step is executed and checked against the cited values.

The contribution of MERIT is to connect table-specific encoder adaptation with evidence-grounded reasoning and executable verification of intermediate steps in one framework.

(2) Neither method has released code or weights, so a controlled comparison is not possible. Their reported numbers also use different evaluation protocols: Qwen3-VL-8B scores 36.01 on TAT-QA in the DiSCo paper but 82.51 under ours. We therefore compare each method's gain over its own backbone, with the backbone score in parentheses:

MethodBackboneWTQTAT-QATabFactInfoTabsHiTab
Table-R1Qwen2-VL-7B+3.80 (46.50)+1.56 (46.50)+66.00 (7.40)*+30.00 (32.80)+25.10 (33.10)
DiSCo + Table-GLSQwen3-VL-8B+7.64 (49.47)+4.53 (36.01)+5.20 (70.21)+27.08 (45.59)−8.12 (35.47)
MERITQwen3-VL-8B+11.82 (61.04)+5.05 (82.51)+12.37 (77.71)+5.59 (73.04)+1.08 (76.90)

* Table-R1 reports 7.40 for its backbone on TabFact.

MERIT achieves the largest reported gains on WTQ and TAT-QA, despite starting from stronger baselines. On TabFact, Table-R1 reports a larger gain (+66.00), but from a much lower baseline (7.40). On InfoTabs and HiTab, the other methods start from baselines 27 to 44 points lower and thus have far more headroom. Moreover, HiTab is part of Table-R1's training data but unseen by MERIT.

W2. Gains mainly from grounded SFT

  1. "The ablations suggest that most of the performance gain comes from grounded SFT, while the proposed specialized components provide only modest additional improvements."
  2. "These results make it unclear how much of MERIT's overall gain is attributable to the proposed encoder adaptation and fine-grained reward design."

We first summarize our reading of Table 4 and then give the details.

  1. SFT accounting for the largest improvement is typical of SFT-then-RL pipelines.
  2. Without encoder adaptation, GRPO is less effective because it receives less training signal.
  3. With encoder adaptation, the lower SFT accuracy arises because the decoder needs more training steps to adapt to the new encoder, while the benefit of RL is fully realized in this regime.

Improvements of SFT. Grounded SFT is not a generic baseline to which the other components are added; it is itself one of MERIT's components. Its training data are our evidence-grounded trajectories, which cite every cell by coordinate and value and are verified for evidence, derivation, and answer. Encoder adaptation, grounded SFT, and decomposed-reward GRPO are designed as one method, so the gain from grounded SFT is part of MERIT's gain. With all three stages, MERIT improves over the backbone by 9.40 points, up from 7.29 points after adapted SFT. That SFT accounts for the largest share is also typical of SFT-then-RL pipelines: for multimodal reasoning, cold-start SFT raises Qwen2.5-VL-3B from 40.00 to 49.46 and GRPO adds only 1.38 points (Wei et al., 2025); Phi-4-reasoning gains 44.7 points on OmniMath from SFT and 5.3 points from RL (Abdin et al., 2025). The RL share in MERIT is larger than in these pipelines.

Response to Reviewer XZ19 (Part 2/3)

W2. Gains mainly from grounded SFT (continued)

Moreover, each stage makes subsequent training more effective. To assess rollout quality before GRPO, we sample eight responses per prompt on 512 RL prompts using the base model and the two SFT checkpoints from Table 4 (Appendix C.6, Table 14):

Starting pointFormat error (%)Zero-variance groups (%)Prompts with no correct rollout (%)pass@8GRPO gain
Base100.0057.2327.1572.85–
SFT, original encoder2.321.9517.1982.81+0.04
SFT, adapted encoder0.811.9515.8284.18+2.11

Before SFT, no response contains all required fields, and 57.23% of GRPO groups receive identical rewards across all 8 rollouts, so GRPO has almost no gradient to learn from. SFT reduces this share to 1.95%, which is what makes GRPO trainable.

Contribution of encoder adaptation and RL. In Table 4, GRPO improves accuracy by only 0.04 points with the original encoder, compared with 2.11 points with the adapted encoder. This indicates that the adapted encoder unlocks the potential of RL, and also confirms the contribution of GRPO. During RL, a higher pass@k provides more opportunities to sample correct trajectories and obtain answer-level reinforcement signals (Yue et al., 2025). With the original encoder, 17.19% of prompts have no correct rollout and therefore receive no positive answer-correctness reward. The adapted encoder reduces this share to 15.82% and raises pass@8 from 82.81 to 84.18, despite having lower SFT accuracy (83.34 vs. 84.46). Thus, adaptation improves the availability of correct trajectories for subsequent GRPO, even without improving the initial SFT accuracy.

This interaction persists when both models start GRPO from comparable SFT accuracies.

  • Setting of the 120-step GRPO runs. To compare GRPO from matched starting points, we use two SFT checkpoints with comparable accuracy: the adapted model trained for one additional SFT epoch (84.39) and the non-adapted SFT model from Table 4 (84.46). From each, we run 120 GRPO steps with the same RL data, hyperparameters, and full reward as in the paper. Due to limited computational resources, this is shorter than the 780 steps used for the final model, so the absolute gains are smaller than those in Table 4.
  • Result. Under this setting, the adapted model gains 0.53 points (95% CI [+0.10, +0.84]), compared with 0.09 points without adaptation.

Beyond RL, adaptation improves table perception from 46.89 to 53.51 in Table 1, and the complete pipeline outperforms the non-adapted one by 0.95 points on average, with gains on each of the five datasets in Table 4.

Decomposed reward. Under the same 120-step setting, removing the evidence or calculation reward reduces the 0.53-point GRPO gain to 0.32 or 0.31 points, which are no longer significant. The process rewards also reduce the share of GRPO groups without gradient from 49.2% under an answer-only reward to 1.95%. Our response to Reviewer 5QpC gives the full ablation (Appendix C.7).

Response to Reviewer XZ19 (Part 3/3)

Q1. Encoder adaptation + SFT versus SFT

  1. "Why does encoder adaptation + SFT underperform SFT without encoder adaptation (83.34 vs. 84.46), while the benefit of encoder adaptation only emerges after GRPO?"

(1) Stage 1 changes the visual features, so the decoder must realign to them during SFT, and the SFT budget in Table 4 is not sufficient. With one additional SFT epoch, the gap closes (84.39 vs. 84.46; Appendix C.6, Table 13). The benefit then emerges after GRPO because adaptation raises the pass@8 ceiling that bounds GRPO, as explained in W2.

Q2. Strict image-only OOD evaluation

  1. "Can the authors also report a strictly image-only OOD evaluation to support the claimed generalization when structured representations are unavailable?"

(1) We removed all textual tables, namely the MinerU Markdown for FinMMR, TableVista, and MMMU-T and the serialized tables for HiTab and FinQA (Appendix C.8, Table 18):

Model (image only)HiTabFinQAFinMMRTableVistaMMMU-TAvg.
Qwen3-VL-8B76.9026.2430.5265.6743.8648.64
MERIT77.9828.2544.6263.7840.3551.00

Without any structured text, MERIT still outperforms the backbone by 2.36 points on average, which is significant with a 95% CI of [+0.70, +3.99], and the FinMMR gain of 14.10 points persists. The MMMU-T gap comes from missing subject knowledge: under the Table 3 setting, adding Wikipedia retrieval closes the gap completely, with both models reaching 45.03, as shown in our response to Reviewer 5QpC (W2 & Q5).

We hope these responses clarify our contributions, and we would be grateful if the reviewer would consider raising the score.

References

  • Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025. arXiv:2504.13837.
  • Lai Wei, Yuting Li, Kaipeng Zheng, Chen Wang, Yue Wang, Linghe Kong, Lichao Sun, and Weiran Huang. Advancing Multimodal Reasoning via Reinforcement Learning with Cold Start. arXiv:2505.22334, 2025.
  • Marah Abdin et al. Phi-4-reasoning Technical Report. arXiv:2504.21318, 2025.
Reviewer_5QpCOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper proposes MERIT, a three-stage framework for multimodal table reasoning that combines table-centric visual encoder adaptation, grounded supervised fine-tuning (SFT), and reinforcement learning with decomposed verifiable rewards. The framework aims to improve visual evidence extraction and reasoning over table images, with serialized tables as optional inputs. Experiments on table question answering and fact verification datasets demonstrate improvements over the Qwen3-VL-8B backbone and several existing baselines, with additional evaluations of out-of-distribution generalization and component ablations.

Critical strengths
  • The paper addresses the limitations of relying solely on serialized tables, particularly the potential loss of structural and visual information. The combination of visual encoder adaptation, grounded SFT, and RL provides a coherent approach to addressing both table perception and reasoning.

  • MERIT achieves improvements over its Qwen3-VL-8B backbone on the five main benchmarks. The paper also includes comparisons with multiple baselines, OOD evaluations, and ablation studies, providing useful evidence of the framework's overall effectiveness.

Critical weaknesses
  • Insufficient justification of the reward design. Although decomposed rewards are a central component of MERIT, their individual contributions are not fully established. The improvement from Answer-only to Full Reward is relatively modest (85.01% to 85.45%, Table 11), and the paper lacks complete leave-one-out ablations and reward-weight sensitivity analysis. The use of a single reference evidence set may also penalize alternative valid reasoning paths.

  • Limited and inconsistent gains from individual components and OOD generalization. Table 4 shows that most of the overall improvement comes from SFT, while encoder adaptation and GRPO provide smaller and less consistent additional gains. Moreover, the OOD improvement in Table 3 is largely driven by FinMMR, with performance degradation on FinQA and MMMU-T. These results make the contributions of individual stages and the broader generalization claims less convincing.

Questions for authors

Questions

  1. Evidence Reward (Lines 337–344, Eq. 10): A Table QA question may have multiple valid sets of supporting evidence. However, the proposed evidence reward measures agreement with a single reference evidence set. Could the authors clarify how alternative but equally valid evidence selections are handled, and whether they might be unfairly penalized?

  2. Reward Component Ablation (Eq. 9, Appendix C.3, Table 11): Although Table 11 provides some reward ablations, the improvement from Answer-only (85.01%) to Full Reward (85.45%) is relatively modest. The contribution of each component remains unclear. It would be helpful to include leave-one-out ablations for the individual reward components, along with a sensitivity analysis of the reward weights in Eq. (9), to better justify their necessity and relative importance.

  3. Necessity of Format Reward (Lines 354–356): Since the SFT stage already explicitly supervises the required output format, it would be useful to report the proportion of responses with formatting errors after SFT and whether this proportion decreases after GRPO. Additionally, an ablation removing the format reward would help establish its necessity.

  4. Reward Weighting (Eq. 9, Appendix A.6): With the current reward weights, the combined contribution of the non-answer rewards can exceed that of the answer reward. This raises the possibility that responses with incorrect final answers could receive higher total rewards than correct ones. Have the authors observed such cases during training, and if so, how are they mitigated?

  5. OOD Generalization (Table 3, Lines 446–451): The OOD improvement appears to be largely driven by FinMMR (+15.65 points), while the model shows performance drops on FinQA (-0.95) and MMMU-T (-4.17). Excluding FinMMR, the average performance across the remaining four datasets is approximately 0.62 points lower than the Qwen3-VL-8B baseline. Could the authors further analyze why the improvement is substantially larger on FinMMR and explain the performance degradation on FinQA and MMMU-T? A more detailed analysis would help clarify the generalization capabilities of the proposed method.

Suggestions:

Several relevant works appear to be missing from the related-work discussion. In particular, TableDART [1] is closely related to the paper's comparison and combination of textual and visual table representations, while SynTab-LLaVA [2] and TABLET [3] are relevant to visual table representation learning and table-specific multimodal training. MMTabReal [4] may also be relevant to the paper's motivation around real-world visual tables. In addition, several evaluated baselines, including InternVL2.5-8B [5], Claude Sonnet 5, and Gemini 3.5 Flash, appear to lack corresponding citations. Please consider checking the related work and baseline references systematically.

[1] X. Xing, W. Yuan, T. Chen, Q. V. H. Nguyen, X. Zhang, and H. Yin, “TableDART: Dynamic Adaptive Multi-Modal Routing for Table Understanding,” International Conference on Learning Representations, vol. 2026, pp. 142584–142609, Apr. 2026.

[2] B. Zhou et al., “SynTab-LLaVA: Enhancing Multimodal Table Understanding with Decoupled Synthesis,” presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 24796–24806. Accessed: Oct. 07, 2026. [Online]. Available: https://openaccess.thecvf.com/content/CVPR2025/html/Zhou_SynTab-LLaVA_Enhancing_Multimodal_Table_Understanding_with_Decoupled_Synthesis_CVPR_2025_paper.html

[3] I. Alonso, I. Miranda, E. Agirre, and M. Lapata, “TABLET: A Large-Scale Dataset for Robust Visual Table Understanding,” International Conference on Learning Representations, vol. 2026, pp. 105851–105887, Apr. 2026.

[4]P. Y. Titiya, J. Trivedi, C. Baral, and V. Gupta, “MMTabReal: Real-World Benchmark for Multimodal Table Understanding,” in Findings of the Association for Computational Linguistics: ACL 2026, M. Liakata, V. P. Moreira, J. Zhang, and D. Jurgens, Eds., San Diego, California, United States: Association for Computational Linguistics, Jul. 2026, pp. 41156–41176. doi: 10.18653/v1/2026.findings-acl.2047.

[5] Z. Chen et al., “Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling,” Sep. 26, 2025, arXiv: arXiv:2412.05271. doi: 10.48550/arXiv.2412.05271.

4 comments

Response to Reviewer 5QpC (Part 1/3)

We thank the reviewer for the detailed and constructive review. We address each concern below with new experiments, all of which have been added to the revised PDF (Appendices C.7 and C.8).

W1(1) & Q1. Alternative valid evidence

  1. "A Table QA question may have multiple valid sets of supporting evidence. However, the proposed evidence reward measures agreement with a single reference evidence set. Could the authors clarify how alternative but equally valid evidence selections are handled, and whether they might be unfairly penalized?"

(1) The evidence reward matches cited cells against the reference set by both coordinate and value, so a valid but different selection can receive a lower evidence score. This does not unfairly affect learning, for two reasons.

First, the evidence reward cannot outweigh answer correctness. In our training rollouts, every correct answer receives a higher total reward than every incorrect answer, as shown in W1(4) & Q4. A lower evidence score therefore only changes the ranking among correct answers and never makes a correct answer lose to an incorrect one.

Second, genuinely different evidence is rare. Among correct rollouts, 16.4% have evidence F1 below 0.5. We inspected 40 of them, labeled by an LLM and verified manually, and none cites irrelevant cells. Most cite the right values but count rows or columns differently, for example whether a title band counts as a row; the rest cite a smaller set of cells that is still sufficient. Matching by value instead of coordinate lowers the share from 16.4% to 8.4% over all 2,834 correct rollouts. Most low scores are thus an indexing mismatch rather than a different line of reasoning.

W1(2) & Q2. Reward ablation and weight sensitivity

  1. "The improvement from Answer-only to Full Reward is relatively modest (85.01% to 85.45%, Table 11) … The contribution of each component remains unclear." (Table 15 in the revised PDF.)
  2. "It would be helpful to include leave-one-out ablations for the individual reward components …"
  3. "… along with a sensitivity analysis of the reward weights in Eq. (9), to better justify their necessity and relative importance."

(1) The answer reward is the primary signal, so the auxiliary rewards are not expected to move average accuracy much. Their main contribution is to provide signal where the answer reward provides none. On 512 RL prompts with 8 rollouts each, 15.82% of groups are all wrong and 33.40% are all correct. With an answer-only reward, all rollouts in these groups receive identical rewards, so 49.2% of prompts give no GRPO gradient. With our full reward, this drops to 1.95%. The evidence reward is the main source of this signal: it separates rollouts in 97.53% of all-wrong groups and 92.98% of all-correct groups.

(2) All variants below start from the same encoder-adapted SFT checkpoint, trained for one additional SFT epoch (84.39%), and run 120 GRPO steps with the same RL data and hyperparameters as in the paper; only the reward differs. Due to limited computational resources, this is shorter than the 780 steps used for the final model, so the gains are smaller than in Table 15 (Appendix C.7, Table 16).

VariantAvg. Acc. (%)Gain over SFT
Without GRPO84.39–
Full reward84.92+0.53 [+0.10, +0.84]
w/o evidence84.71+0.32, n.s.
w/o calculation84.70+0.31, n.s.
w/o format84.94+0.55
All auxiliary ×0.5 / ×284.82 / 84.86+0.43 / +0.47
Evidence ×0.5 / ×284.97 / 84.85+0.58 / +0.46
Calculation ×0.5 / ×284.95 / 84.86+0.56 / +0.47
Format ×0.5 / ×284.68 / 84.63+0.29 / +0.24

With the full reward, the gain over SFT is significant. Removing the evidence or calculation reward reduces it to 0.32 or 0.31 points, which are no longer significant, so both rewards contribute. The format reward is discussed in W1(3) & Q3.

(3) All twelve configurations end above the SFT starting point. Halving or doubling the evidence weight, the calculation weight, or all auxiliary weights together changes average accuracy by at most 0.10 points. MERIT therefore does not depend on careful weight tuning.

Response to Reviewer 5QpC (Part 2/3)

W1(3) & Q3. Necessity of the format reward

  1. "It would be useful to report the proportion of responses with formatting errors after SFT and whether this proportion decreases after GRPO. Additionally, an ablation removing the format reward would help establish its necessity."

(1) We measure format errors using eight rollouts for each of the 512 RL prompts, starting from the same SFT checkpoint used in the reward ablations in W1(2) & Q2 (Appendix C.7, Table 17):

ModelFormat errorTruncation
After SFT (GRPO start)0.56%0.17%
GRPO, full reward0.15%0.10%
GRPO, w/o format reward0.27%0.17%

With the format reward, GRPO reduces format errors by 73%, compared with 52% without it, and also reduces truncation, which stays unchanged without it. We agree that SFT already establishes the format, so the format reward is a safeguard rather than a source of accuracy: removing it leaves accuracy essentially unchanged (84.94 vs. 84.92). It is still needed because a malformed response cannot be parsed and therefore receives no evidence or calculation feedback.

W1(4) & Q4. Reward weighting

  1. "With the current reward weights, the combined contribution of the non-answer rewards can exceed that of the answer reward … Have the authors observed such cases during training, and if so, how are they mitigated?"

(1) We have not observed such cases. On 512 RL prompts with 8 rollouts each, every correct-answer rollout scores at least 0.83 and every incorrect one at most 0.60. No incorrect answer outranks a correct one, so no mitigation is needed.

W2(1). Gains mainly from SFT

  1. "Table 4 shows that most of the overall improvement comes from SFT, while encoder adaptation and GRPO provide smaller and less consistent additional gains."

(1) It is typical of SFT-then-RL pipelines that SFT provides most of the gain (Guo et al., 2025; Lambert et al., 2025), and in MERIT, SFT also establishes the output format on which the process rewards rely. As analyzed in our response to Reviewer ZaSc (W2 & Q2), GRPO together with encoder adaptation raises the upper limit of SFT.

  • Without encoder adaptation, GRPO is less effective because it receives less training signal.
  • With encoder adaptation, the lower SFT accuracy arises because the decoder needs more training steps to adapt to the new encoder, while the benefit of RL is fully realized in this regime.

Please see our response to Reviewer ZaSc (W2 & Q2) and Appendix C.6 for details.

W2(2) & Q5. OOD generalization

  1. "Could the authors further analyze why the improvement is substantially larger on FinMMR and explain the performance degradation on FinQA and MMMU-T?"

(1) The OOD gains are largest where the image carries the most information, and the two drops have causes outside table reasoning (Appendix C.8).

  • FinMMR. Its tables are visually complex and noisy and often span multiple images, so evidence grounded in the image corrects the most errors. The gain holds without any parsed text: with image-only input, MERIT improves FinMMR by 14.10 points, as shown in our response to Reviewer XZ19 (Q2).
  • FinQA. The −0.95 drop appears only with FinQA's official serialized table, an input MERIT never sees in training. This table is accurate, so the image adds little (+1.73). Under image-only input, MERIT improves FinQA from 26.24 to 28.25 (response to Reviewer XZ19, Q2), and with identical gold facts its accuracy rises by 23.94 points (response to Reviewer ZaSc, W1(1)). The drop thus reflects an unseen input format, not weaker table reasoning.
  • MMMU-T. These university exam questions mainly test subject knowledge. SFT can cause the model to forget knowledge unrelated to the training tasks (Jin et al., 2025), which lowers performance. When both models are given a Wikipedia retrieval tool under the same setting as Table 3, the gap closes completely (Appendix C.8, Table 19):
ModelWithout retrieval (Table 3)With retrieval
Qwen3-VL-8B42.7745.03 (+2.26)
MERIT38.6045.03 (+6.43)

The drop therefore comes from missing subject knowledge rather than weaker table reasoning.

Response to Reviewer 5QpC (Part 3/3)

Suggestions. Missing references

Thank you. We have added TableDART, SynTab-LLaVA, TABLET, and MMTabReal to the related work (Section 2), and cited InternVL2.5-8B, Claude Sonnet 5, and Gemini 3.5 Flash in the experimental setup.

We hope these results address the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score.

References

  • Daya Guo, Dejian Yang, Haowei Zhang, et al. DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning. Nature, 645:633–638, 2025.
  • Nathan Lambert, Jacob Morrison, Valentina Pyatkin, et al. Tülu 3: Pushing Frontiers in Open Language Model Post-Training. COLM 2025. arXiv:2411.15124.
  • Hangzhan Jin, Sitao Luan, Tianwei Ni, Sicheng Lyu, Guillaume Rabusseau, Reihaneh Rabbany, Doina Precup, and Mohammad Hamdaqa. RL Fine-Tuning Heals OOD Forgetting in SFT. arXiv:2509.12235, 2025.

Thank you for the detailed explanations. My concerns and questions have been addressed, and I will raise my score.

Reviewer_ZaScOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies question answering and fact verification over table images and proposes a three-stage training framework, MERIT. It first adapts the visual encoder through table reconstruction, cell localization, and color recognition. It then applies supervised fine-tuning using trajectories with explicit cell evidence and calculations, followed by GRPO with separate rewards for evidence, calculation, answers, and output format. Experiments cover five in-distribution datasets and five additional datasets, with ablations on input modalities, training stages, and reward components. With image-only input, the complete method improves average accuracy over the Qwen3-VL-8B backbone by 9.40 percentage points and demonstrates some cross-dataset transfer.

Critical strengths
  1. The method offers a coherent training pipeline for a practically relevant problem. Understanding table images requires handling dense text, two-dimensional structure, evidence selection, and calculation. The paper designs supervision for these components and uses explicit cell coordinates and values to make generated solutions more directly verifiable. This provides a useful approach to building reliable table-reasoning systems.

  2. The overall gains are clear, and the component analyses are informative. With image-only input, average accuracy increases from 76.05% to 85.45%; on five additional datasets, average accuracy increases from 52.27% to 54.90%. Ablations of training stages, adaptation objectives, and rewards also reveal limitations, including the negligible standalone QA benefit of encoder adaptation and the modest incremental gains from intermediate rewards. These results help assess the practical contribution of individual components.

Critical weaknesses
  1. The sources of the gains and the conditions under which they arise are not yet sufficiently established. Figure 3(a) supports a bottleneck in visual evidence acquisition, but it compares different encoders and therefore does not directly quantify the change in reasoning ability before and after MERIT training. The error analysis reports fewer reasoning errors, but attribution primarily relies on an LLM classifier, and examples may move between error categories. Furthermore, the modality ablation establishes that images provide complementary information without directly separating recovery from parsing errors from other benefits, such as preserving layout. This limits the interpretation of the mechanism and practical applicability of the method.

  2. The interactions among training stages need stronger explanation and evidence of stability. In Table 4, adding GRPO after SFT improves average accuracy by only 0.04 percentage points without visual adaptation, but by 2.11 points with adaptation. Meanwhile, visual adaptation followed by SFT performs worse than SFT alone. The complete pipeline improves over SFT + GRPO by 0.95 points, and the full reward improves over answer-only reward by 0.44 points. Whether these interactions are stable or sensitive to training settings and run-to-run variation directly affects the assessment of the three-stage design and the specific reward formulation.

Questions for authors
  1. Separating perception and reasoning gains. Could the authors compare the backbone and final MERIT on the same evaluation subset while providing both with identical Gold Facts? Please also clarify the source, exact content, and presentation of these facts. This would help establish whether training yields clear reasoning gains when evidence acquisition is bypassed.

  2. Stability of the interactions among training stages. How do the authors explain why visual adaptation followed by SFT underperforms SFT alone, while adaptation substantially increases the benefit of subsequent GRPO? Do these ablations use identical data, training budgets, and checkpoint-selection rules? If repeated runs are available, please report variability for the key combinations; otherwise, a limited repeatability check would help establish the stability of the claimed complementarity.

  3. Direct evidence of recovery from parsing errors. On an existing evaluation subset with original structured tables, could the authors provide paired comparisons of original-table text, parsed text, and each representation augmented with the table image? An analysis distinguishing content-recognition errors from structural-association errors would help identify when images are beneficial. The goal is to validate the paper’s existing central motivation, rather than to request a new large-scale benchmark.

4 comments

Response to Reviewer ZaSc (Part 1/3)

We thank the reviewer for the positive assessment and the constructive suggestions. We ran all the requested analyses and address each point below. All new results have been added to the revised PDF (Appendices C.1–C.3, C.6, and C.7).

W1(1) & Q1. Separating perception and reasoning gains

  1. "Figure 3(a) supports a bottleneck in visual evidence acquisition, but it compares different encoders and therefore does not directly quantify the change in reasoning ability before and after MERIT training."
  2. "Could the authors compare the backbone and final MERIT on the same evaluation subset while providing both with identical Gold Facts? Please also clarify the source, exact content, and presentation of these facts."

(1) Figure 3(a) is intended to identify the bottleneck rather than measure MERIT's reasoning gains. The two encoders share the same NaViT architecture, Qwen2-VL initialization, and decoder; MinerU 2.5 undergoes additional document-oriented training. The performance gap therefore reflects differences in encoder training. MERIT's perception gains on the same backbone are reported in Tables 1 and 12 (Table 10 in the original submission), including gains of +8.10 in cell extraction and +10.12 in cell localization. To assess reasoning gains separately, we conduct the gold-facts comparison in (2).

(2) The gold facts are derived from the official evidence annotations of each dataset and expressed as short natural-language sentences (e.g., "the 2015 net revenue ... is $5,829"). They contain neither final answers nor calculation steps. Both models receive identical prompts within each input setting. On FinQA, we use the table-only questions from Figure 3(a). On HiTab, we restrict the comparison to arithmetic questions, since the annotated evidence cell directly contains the answer to lookup questions. All rows below use these subsets, so their scores differ from the full-test-set results elsewhere in our responses (Appendix C.2, Table 10).

DatasetInputQwen3-VL-8BMERITΔ (95% CI)
FinQAImage only40.9341.22+0.28 [−2.69, +3.12]
FinQAGold facts only48.8772.80+23.94 [+20.25, +27.62]
FinQAGold facts + image67.4272.80+5.38 [+2.27, +8.36]
HiTabImage only46.4960.00+13.51 [+5.95, +21.08]
HiTabGold facts only59.4674.05+14.59 [+8.11, +21.08]
HiTabGold facts + image69.1976.22+7.03 [+0.54, +14.05]

With identical gold facts, visual evidence retrieval is bypassed, yet MERIT still outperforms the backbone by 23.94 points on FinQA and 14.59 points on HiTab. Both confidence intervals exclude zero, indicating that MERIT's gains are not solely attributable to visual evidence acquisition. When the image is also provided, MERIT retains statistically significant advantages of 5.38 and 7.03 points, respectively.

W1(2). Reliance on an LLM classifier

  1. "The error analysis reports fewer reasoning errors, but attribution primarily relies on an LLM classifier, and examples may move between error categories."

(1) We validated the classifier in two ways. Human reviewers agreed with the LLM labels on 91.7% of 48 randomly sampled error cases (Cohen's κ = 0.88). An independent classifier based on GPT-5.5 shows consistent trends, including a reduction in localization errors from 27 to 18. Moreover, the total number of errors decreases from 405 to 290 regardless of categorization, and the number of errors decreases in every category (Figure 5). Finally, MERIT preserves over 97% of the examples that the backbone answers correctly, so the reduction is not offset by new errors (Appendix C.1).

Response to Reviewer ZaSc (Part 2/3)

W1(3) & Q3. What the image recovers

  1. "Could the authors provide paired comparisons of original-table text, parsed text, and each representation augmented with the table image?"
  2. "The modality ablation establishes that images provide complementary information without directly separating recovery from parsing errors from other benefits, such as preserving layout."
  3. "An analysis distinguishing content-recognition errors from structural-association errors would help identify when images are beneficial."

(1) We evaluate MERIT on the same questions with four table inputs, where the MinerU-parsed table is obtained by converting the table image into text. We use HiTab and FinQA, the two datasets that provide serialized tables (Appendix C.3, Table 11):

Table inputHiTabFinQA
Official serialized table43.6545.87
Official serialized table + image68.9147.60
MinerU-parsed table66.4624.41
MinerU-parsed table + image69.5128.16

Textual tables alone are often unreliable, and adding the image as auxiliary information improves both HiTab and FinQA, whether the textual table is the official serialization or produced by parsing. This supports our motivation.

(2) The image provides two separable benefits (Appendix C.3, Figures 6 and 7).

Recovering parsing errors. Adding the image to the parsed table fixes 122 parsing-induced errors on HiTab; on FinQA, it fixes 60 and breaks 17. In the example below, MinerU drops the header corner cell and the 2004 values, so the 2005 values fall under "2004":

Image200620052004
Sales$4,925$4,625$4,545
Operating profit$399$219$373
MinerU output: 200620052004
Sales$4,925$4,625
Operating Profit$399$219

Asked for the 2004 profit margin, MERIT computes 219 / 4,625 = 4.74% from the parsed table alone, but the correct 373 / 4,545 = 8.21% once the image is added.

Preserving layout. Even without parsing errors, HiTab's official serialization flattens multi-level headers, producing strings such as "eastern ontario / french-language workers", and can misalign section rows. In the example below, the empty section row "household income quartile" is merged with the first quartile, which shifts every value down by one row:

RowImageSerialized table
quartile 1 (lowest)70.2–
quartile 281.870.2
quartile 388.881.8
quartile 4 (highest)93.588.8

Asked for the top quartile, MERIT answers 88.8 from the serialized table and 93.5 with the image. Across HiTab, adding the image to the serialized table raises accuracy from 43.65 to 68.91.

(3) In the cases we inspected, most fixes correct structural-association errors rather than content-recognition errors. Both examples above follow this pattern: the values are read correctly but attached to the wrong column or row. Images are therefore most beneficial when the textual table misaligns values and headers, whether through parsing or through serialization.

Response to Reviewer ZaSc (Part 3/3)

W2 & Q2. Stability of stage interactions

  1. "Visual adaptation followed by SFT performs worse than SFT alone."
  2. "Adding GRPO after SFT improves average accuracy by only 0.04 percentage points without visual adaptation, but by 2.11 points with adaptation."
  3. "The complete pipeline improves over SFT + GRPO by 0.95 points, and the full reward improves over answer-only reward by 0.44 points. Whether these interactions are stable or sensitive to training settings and run-to-run variation directly affects the assessment of the three-stage design and the specific reward formulation."
  4. "Do these ablations use identical data, training budgets, and checkpoint-selection rules? If repeated runs are available, please report variability for the key combinations."

(1) Stage 1 changes the encoder's representations, so the decoder needs more SFT steps to realign to them, and the SFT budget in Table 4 is not sufficient. With one additional SFT epoch, the adapted model catches up with the non-adapted one (Appendix C.6, Table 13):

ModelAvg. Acc. (%)
Non-adapted SFT84.46
Adapted SFT, Table 4 budget83.34
Adapted SFT, +1 epoch84.39

(2) A higher pass@k provides more opportunities to sample correct trajectories and obtain answer-level reinforcement signals (Yue et al., 2025). For the SFT checkpoints, 17.19% of prompts have no correct answer among eight rollouts with the original encoder, compared with 15.82% with the adapted encoder, even though the latter has lower SFT accuracy (83.34 vs. 84.46) (Appendix C.6, Table 14). Encoder adaptation is therefore important for our pipeline: it raises the upper limit of RL by providing more training signal. This is reflected in Table 4, where GRPO raises the adapted SFT model from 83.34 to 85.45.

Setting of the 120-step GRPO runs. To compare GRPO from matched starting points, we use two SFT checkpoints with comparable accuracy: the adapted model trained for one additional SFT epoch (84.39) and the non-adapted SFT model from Table 4 (84.46). From each, we run 120 GRPO steps with the same RL data, hyperparameters, and full reward as in the paper. Due to limited computational resources, this is shorter than the 780 steps used for the final model, so the absolute gains are smaller than those in Table 4. Under this setting, the adapted model gains 0.53 points (95% CI [+0.10, +0.84]), compared with 0.09 points for the non-adapted model, consistent with the trend in Table 4.

(3) Both gains are consistent rather than noise. The 0.95-point gain of the complete pipeline follows from (2) and holds on all five datasets in Table 4. For the reward, we repeat the 120-step setting from the adapted checkpoint (84.39) with different reward variants. The full reward yields the significant 0.53-point gain above, whereas removing the evidence or calculation reward reduces it to 0.32 or 0.31 points, which are no longer significant. The process rewards also provide training signal that the answer reward cannot: they reduce the share of GRPO groups without gradient from 49.2% to 1.95%. Our response to Reviewer 5QpC gives the full ablation (Appendix C.7).

(4) All ablations in Table 4 use identical data, training budgets, and checkpoint-selection rules. For this response, we re-ran the key combinations several times and obtained consistent results. The interactions are also robust to training settings: the same pattern appears under the longer SFT budget in (2), and all twelve reward-weight configurations in our response to Reviewer 5QpC improve over SFT.

We hope these results address the reviewer's concerns, and we would be grateful if the reviewer would consider them in the final assessment.

Reference

  • Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? NeurIPS 2025. arXiv:2504.13837.

Thank you for your detailed response, which has addressed my concerns. As my original rating was already positive, I will maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.