acceptodds
Under review as a conference paper at ICLR 2027

TableMoE: Reinforcement Learning to Route Connector Experts for Multimodal Table Reasoning

Abstract

Tables are not just pictures of numbers; they are structured objects where rows, headers, units, and grouping all matter. We ask whether a model can learn to combine different table views through answer-level feedback, instead of committing to one rigid representation. TableMoE keeps a pretrained visual connector and adds experts aligned to HTML, JSON, and table-redrawing code. At each visual token, the router selects two experts and mixes their outputs. A reward-guided training stage then adapts the router, experts, and language-model adapters together. Across six benchmarks, this adaptation improves average accuracy by 4.14 percentage points over dense softmax SFT, with a similar gain from the self-critical variant. Analysis of the learned mixture reveals a dominant General expert with spatially varying companions, while a fixed General+JSON pair retains most of the learned routing accuracy on the development subset. These findings support reward-based adaptation of the aligned mixture, but provide more limited evidence that spatially adaptive expert selection is necessary for accurate answers.

Primary area:
reinforcement learning
Keywords:
Multimodal Table Reasoning, Mixture-of-Connector Experts, Expert Routing, Reinforcement Learning, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZEGrOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TableMoE, a multimodal table-reasoning framework that combines four connector experts: a pretrained General expert and three experts aligned to HTML, JSON, and table-redrawing code. A token-level router selects and fuses two experts for each visual token, and the mixture is further adapted using answer-level reinforcement learning with Multi-GRPO and SC-RPO. The experiments show consistent aggregate improvements over the matched SFT baselines, while the routing analysis suggests that the General expert is dominant and that a fixed General+JSON pair retains much of the learned-routing performance.

Critical strengths
  • Reward-based adaptation improves the matched SFT baselines across all six main benchmarks. Both RL variants use the same four-expert architecture as Det-SFT and Soft-SFT, making these comparisons informative for assessing the contribution of answer-level reward adaptation.

  • The paper includes several analyses of the learned routing behavior. Spatial routing visualizations, expert-exclusion experiments, and fixed-pair interventions provide information beyond aggregate benchmark accuracy and help distinguish gains from adapting the expert mixture from the contribution of individual spatial routing decisions.

Critical weaknesses
  • The incremental value of fine-grained token-level adaptive routing is not convincingly established. A fixed General+JSON pair retains most of the learned-routing performance on Dev300, despite changing a large fraction of token-level expert assignments. Since token-level spatial routing is a central architectural component of TableMoE, the absence of a simpler adaptive baseline, such as selecting one expert pair per image, makes it difficult to determine whether token-level routing itself provides a meaningful advantage.
Questions for authors
  • Why is Top-2 routing used throughout the model? What motivates selecting exactly two experts for each visual token?

  • Given that a fixed General+JSON pair retains most of the learned-routing accuracy, does token-level adaptive routing provide a measurable advantage over a simpler image-level adaptive routing strategy that selects one expert pair per image?

  • Do different datasets show different preferences for particular experts, and do these preferences relate to differences in table structure or task characteristics?

  • Several writing and implementation details would benefit from clarification:

    • Several baselines and datasets appear to be missing their corresponding citations. In particular, a number of literature-reported methods in Table 1, including SynTab-LLaVA, CoReTab, MM-Table-R1, Table-R1, HIPPO, Turbo, and TableDART, do not appear to have corresponding references. The Qwen2.5-VL backbone and TableVQA-Bench also seem to be uncited.
    • Line 134: What is the motivation for using a bias-free router, zi=Wrxi∈R4z_i = W_r x_i \in \mathbb{R}^4, rather than including a bias term?
    • Line 144: How are the routed visual representations (h1,…,hV)(h_1, \ldots, h_V) incorporated with the question tokens into the language model input? For example, are they concatenated in the input sequence or combined in another way?
    • Line 146: What is the intuition behind sampling an ordered pair without replacement during training? Is this primarily intended to introduce exploration during RL training, since deterministic Top-2 routing would otherwise produce no routing variation?
Reviewer_V7z3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the problem of Table Question Answering (TableQA). It proposes TableMoE, a mixture-of-experts-based method that includes a pretrained visual connector with experts aligned to HTML, JSON, and table redrawing code. To optimize the router, the authors propose Multi-GRPO, a method that requires the model to explore several routing trajectories and use the resulting rewards to update the router, experts, and adapters. The resulting design has been shown to outperform self-supervised fine-tuning.

Critical strengths
  • The paper addresses an important problem. Table recognition serves as an important component of digitization.
  • The analysis of TableMoE is extensive, giving readers a better understanding of its properties and design choices (although comparisons with existing methods remain limited, as discussed below).
  • Overall, the presentation is clear, and the paper is well-written.
Critical weaknesses
  • The paper seems outdated. The literature cited is mostly from 2024 and 2025. More importantly, the model used in the experiments is Qwen2.5-VL-7B-Instruct. It is unclear whether newer pretrained models can already outperform TableMoE.
  • The table reasoning results are not impressive (Table 1). The results show that existing alternatives (e.g., TableMix and MM-Table-R1-3B) can already achieve better performance than TableMoE, raising a serious question about whether the proposed method still makes a useful contribution to the field.
  • The unique contribution is unclear. With MoE architectures widely adopted across various tasks, the novelty of applying MoE to TableQA and the challenges involved have not been clearly articulated.
Questions for authors
  1. How does TableMoE perform compared with more recent TableQA-specific methods?
  2. How do more recent pretrained models perform on the evaluated task? Do their improvements surpass those achieved by TableMoE? Can TableMoE further strengthen more recent models (e.g., by replacing Qwen2.5 with a more up-to-date option)?
  3. Could the authors elaborate on the quantitative results in Table 1 and clarify the improvements achieved?
  4. Could the authors summarize the unique contributions and challenges involved in applying MoE to TableQA?
Reviewer_9uD1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The contributions are threefold: (1) the construction of a multi-expert table connector that preserves pre-trained visual signals while being aligned for HTML/JSON/Code formats; (2) the demonstration that reward-based adaptation, applied to this same four-expert architecture, outperforms both Det-SFT and Soft-SFT (based on the average across six benchmarks); and (3) an analysis of routing usage—conducted via routing space visualization, expert exclusion, and the fixing of specific expert pairs—which reveals that the "General" expert is utilized frequently, though its exclusion effect varies depending on the dataset.

Critical strengths

The paper analyzes "the benefits brought by hybrid adaptation" and "whether each spatial routing decision is necessary" separately, and finds out the intervention through expert exclusion and fixed experts

The experiment disclosure is relatively complete: SFT baseline under the same four-expert architecture, six benchmarks plus TableVQA-Bench, reward baseline ablation, parser correction instructions and a large number of appendix configurations

Critical weaknesses

The causal attribution is insufficient. During the RL phase, the router, experts, and LoRA modules are updated simultaneously; since the SFT baseline also includes updates to the experts and adapters, it is impossible to determine whether the performance gains observed across the six benchmarks stem from token-level adaptive routing or simply from the fine-tuning of the experts and LoRA modules. This directly undermines the paper’s claimed contribution.

The evidence supporting the necessity of routing and its generalizability is weak. The comparisons against fixed routing configurations are based on only 300 development questions and a single checkpoint, with the "optimal" fixed configuration selected using the same subset used for evaluation. Furthermore, TableVQA-Bench shares its data source with the training/alignment set, and the experiments are limited to a single backbone and a single random seed. Consequently, the conclusions regarding the utility of adaptive routing and its cross-task generalization are not robust; the current evidence is insufficient to substantiate a claim that such routing is necessary.

Questions for authors

Could you provide additional ablation studies that match the training budget—specifically: updating only the router, updating only the experts/LoRA, and applying RL while keeping the "General + JSON" components frozen—and report the average performance across the six benchmarks for each? This would allow for a direct quantification of the contribution of learned routing to the overall performance gains.

Could you select a fixed set of expert pairs using an independent validation set, test them on held-out questions, and report the performance ranges—ideally across multiple seeds and using at least a second backbone model? This would help clarify the strength of the conclusion that "fixed pairs approximate learned routing."

Could the main benchmarks and TableVQA-Bench be re-evaluated using a corrected multi-value parser? If not, is it possible that the uncorrected parser might alter the task rankings or the conclusions drawn regarding RL versus SFT?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions (Updated Manuscript)

We have revised the manuscript to incorporate the additional experiments and clarifications discussed during rebuttal.

The main updates are:

  • Title and contribution scope: Retitled the paper TableMoE: Reward-Guided Adaptation of Routed Connector Experts for Multimodal Table Reasoning and refined the abstract and contributions to distinguish mixture adaptation from router-specific gains.
  • Expanded results (§4.2–4.3; Appendix K): Added extended-training, Qwen3-VL-8B, and additional transfer evaluations.
  • Routing analyses (§4.5): Added parameter-freezing controls, independent fixed-pair evaluation across three Qwen2.5 seeds and a second backbone, image-level deployment comparisons, and structural-stratum analyses.
  • Scoring audit (Appendices H and K.6): Restricted multi-value parsing to TQA while retaining task-specific classification matchers, and synchronized the affected tables and text. The RL-over-SFT conclusion remains unchanged.
  • Clarity and reproducibility: Clarified SC-RPO, backbone-specific connector implementations, training configurations, and limitations; supplemented citations and improved table layout.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.