acceptodds
Under review as a conference paper at ICLR 2027

TWT: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoning

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated remarkable reasoning capabilities across modalities such as images and text. However, tabular data, despite being a critical real-world modality, remains relatively underexplored in multimodal learning. In this paper, we focus on the task of Tabular-Vision Multi-Modal Understanding (TVMU) and identify three core challenges: (1) high structural variability and data incompleteness in tables, (2) implicit and complex feature dependencies, and (3) significant heterogeneity in problem-solving pipelines across downstream tasks. To address these issues, we propose Thinking with Tables (TWT). TWT employs a program-aided code-based neuro-symbolic reasoning mechanism that facilitates key operations, such as information extraction and element modeling, by interacting with external environments. We evaluate TWT on eight representative datasets. Experimental results demonstrate that TWT consistently outperforms existing baselines by an average of 10% in accuracy, achieving performance comparable to, or even surpassing, proprietary commercial SOTA LLMs on TVMU tasks.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Multi-Modal Reasoning, Neuro-Symbolic Reasoning, Tabular Data

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cAjZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper provides a systematic analysis of the tabular-vision multi-modal understanding task and propose Thinking with Tables, a framework that utilizes a code-based interactive reasoning mechanism to model the complex tabular information.

Critical strengths
  • The paper proposes a program-aided, interactive reasoning mechanism that bridges multimodal inputs, successfully handling structural variability and data incompleteness.
  • The proposed TWT combines TO-SFT to internalize diverse task pipelines with AL-GRPO to optimize execution stability and reasoning convergence.
  • TWT consistently outperforms existing open-source baselines across diverse tasks and datasets.
Critical weaknesses
  • TWT is a method designed for the TVMU task which has a high degree of domain specificity. TWT is not evaluated on broad foundational benchmarks, leaving its generalizability beyond specialized tabular-visual domains unverified.
  • The training pipeline depends heavily on synthesizing reasoning traces using advanced proprietary or larger models, which introduces teacher bias and limits self-contained scalability.
Questions for authors
  • The paper brands TWT as a neuro-symbolic reasoning framework, yet the symbolic component is essentially standard Python code generation executed in a sandbox via code interpreters. How does TWT genuinely incorporate symbolic reasoning beyond standard tool-aided code execution? Is the term "neuro-symbolic" overstated for what is fundamentally a code-interpreter agent loop?
  • Given that TWT is specifically trained and optimized on synthesized domain-specific trajectories (via TO-SFT and AL-GRPO), to what extent is the performance comparison against zero-shot or general-purpose MLLMs entirely fair? Would TWT retain its performance advantage if general baseline models were also given access to equivalent in-domain supervised fine-tuning?
  • The framework relies heavily on iterative interactions with a Python sandbox environment, but what happens when the model generates syntactically correct code that contains semantic logic errors or hallucinations regarding tabular data structures?
  • The paper lacks an evaluation of computational efficiency (latency, multi-turn sandbox overhead, and token cost) as well as broader generalization benchmarks. Could the authors provide a comprehensive analysis of end-to-end inference latency under multi-turn code execution and report performance on general multimodal benchmarks?
4 comments

We thank the reviewers for the valuable review.

W1

This paper proposes a task framework for tabular understanding. The same framework and interaction protocol cover four question-answering datasets and four prediction datasets spanning classification and regression. The differences across question answering, classification, and regression constitute precisely the heterogeneity of task procedures that this paper aims to address, rather than designing a solver for a single dataset.

Generalization ability is a valuable question discussed in our conclusion. For general multimodal benchmarking, we additionally conduct experiments on the MMMU dataset and compare TWT with tabular-related open-source MLLMs. The results are as follows:

ModelMMMU
Table-LLaVA17.68
HIPPO35.98
TWT65.29

The experimental results show that TWT not only achieves superior performance on tabular tasks but also exhibits strong general capabilities.

W2

Training does not stop at imitating teacher trajectories. After TO-SFT, AL-GRPO constructs rewards using the final task answers and continues optimizing the model. Table 2 shows the additional gains of this stage over SFT. In Table 1, the 8B TWT model also outperforms Qwen3-VL-Plus on six of the eight tasks. These results indicate that the final system’s task performance is not simply equivalent to directly invoking the teacher model.

The performance gains of our method come from supervised trajectories filtered by their outcomes establishing initial capabilities, followed by task rewards and environment interactions further optimizing the policy. Teacher bias may remain in the initial data, which is a limitation of the data-generation method that we will discuss further.

Q1

Symbolic reasoning does not necessarily require a theorem prover or a dedicated logical language. In the era of large language models, reasoning is mainly enhanced through three symbolic paradigms. Our work follows the program-aided paradigm: the neural model translates natural-language intent into explicit variables, conditions, filtering, and computational operations, which are then executed by an interpreter according to program semantics.

Compared with generating answers solely in natural language, this process converts part of the reasoning into inspectable and executable structured operations. The contribution of this paper is to address TVMU through a neuro-symbolic (NeSy) system that unifies table question answering and prediction procedures and achieves strong performance.

Reference: Xiao-Wen Yang, Jie-Jing Shao, Lan-Zhe Guo, Bo-Wen Zhang, Zhi Zhou, Lin-Han Jia, Wang-Zhou Dai, and Yu-Feng Li. Neuro-symbolic artificial intelligence: Towards improving the reasoning abilities of large language models. IJCAI Survey.

Q2

First, the general MLLMs receive the same resources, making the comparison fair in terms of the evaluation protocol. TWT’s ability to outperform models with more parameters itself demonstrates the effectiveness of our proposed fine-tuning method.

Second, regarding tabular models, they have all been trained using table-related domain knowledge, have their own training methods for solving tabular problems, and may even use larger amounts of training data.

To further illustrate this point, taking table QA as an example, we evaluate these models following their training and testing settings. Some tabular models, even when provided with complete tables, perform worse than TWT without complete tables in its input: Table-LLaVA achieves 16.62 on WTQ and 53.20 on TabMWP, while HIPPO achieves 85.83 on WTQ and 49.10 on TabMWP. Together with the results in “Comparison to Tabular/Image Baselines,” TWT remains the best overall on this class of tasks.

We fine-tune InternVL3.5-8B and obtain the following results:

MethodTabMWP ↑WikiTQ ↑FinQA ↑TAT-QA ↑Adoption ↑SkinCA ↑Pawpularity ↓Paintings ↓
TWT (Ours)97.1078.9561.9572.8232.9073.650.1990.093
InternVL3.8-8B95.1076.8560.9571.2831.5069.780.02060.102

TWT achieves better performance overall. Meanwhile, the performance improvements of the InternVL model also provide strong evidence for the effectiveness of our training pipeline.

Q3

TWT performs subsequent reasoning based on code execution outputs. When hallucinations occur, particularly hallucinations about table structures, the returned code execution results may differ from the model’s expectations. Subsequent reasoning therefore provides an opportunity to correct these errors.

Q4

Regarding computational efficiency, we separately measure latency, sandbox overhead, and token usage for QA and prediction tasks.

TypeLatencySandbox overheadTokensLLM calls
Table QA26.08 s2.17 s12393.22
Table Prediction102.54 s4.46 s378637.06

For general multimodal benchmarking, results are shown in W1. The experimental results show that TWT not only achieves superior performance on tabular tasks but also exhibits strong general capabilities.

Thanks for addressing my questions. However, I still have reservations about the theoretical innovation and practical value of the method, so I maintain my score.

Thank you for your follow-up and for the time and effort you have devoted to reviewing our work. We appreciate your consideration of our responses. To better understand your remaining reservations about “theoretical innovation” and “practical value,” we would appreciate clarification on which specific concerns remain unresolved and why our explanations have not sufficiently addressed them.

Regarding innovation, your original question concerned the substance of the neuro-symbolic characterization relative to standard code-interpreter agents. If this concern remains, could you please identify which claimed distinction our response failed to establish? If “theoretical innovation” refers to an additional concern beyond the original question, it would be helpful to understand which specific contribution you find insufficient and the reasoning behind that assessment, so that we can respond meaningfully.

Regarding practical value, your original review kindly acknowledged TWT’s ability to handle structural variability and data incompleteness, together with its consistent improvements over open-source baselines across diverse tasks and datasets. We believe these strengths provide concrete evidence of utility within the intended application setting. Could you please clarify how these acknowledged benefits are weighed against your remaining concerns about generalization, scalability, reliability, or efficiency, and which aspects of our responses you find insufficient?

We would therefore be grateful if you could reconsider the assessment in light of our responses and clarify any remaining grounds for the current score. Connecting the broader reservations to specific unresolved issues would help us better understand your assessment and allow for a more constructive discussion of the paper’s contributions and limitations. Thank you again for your time and engagement.

Reviewer_eXBaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on Tabular-Vision Multi-Modal Understanding (TVMU), covering both multimodal table question answering and multimodal table prediction tasks. To handle complex table structures, feature dependencies, and task differences, the authors propose Thinking with Tables (TWT), a framework that interacts with an external Python sandbox environment via code generation. TWT is trained on Qwen3-VL-8B using a two-stage strategy: Task-Oriented Supervised Fine-Tuning (TO-SFT) on synthesized multi-turn reasoning trajectories, followed by Adaptive Loss-Scaled GRPO (AL-GRPO) to improve answer accuracy and code execution stability. Experiments across eight datasets show that TWT outperforms open-source baselines and achieves competitive results compared to commercial models.

Critical strengths
  1. The paper attempts to unify two distinct tasks: multimodal table question answering and multimodal table prediction under a single code-aided agentic framework, which is a practical direction for versatile tabular data analysis.

  2. The authors evaluate their method across eight diverse datasets covering both classification/regression and QA tasks. The ablation studies confirm that combining supervised fine-tuning (TO-SFT) with reinforcement learning helps reduce code execution errors and non-convergence rates.

Critical weaknesses
  1. Although the paper emphasizes visual challenges like scanned documents, layout degradation, and complex table structures, the actual setup bypasses visual table parsing by directly providing clean, complete CSV files in the sandbox environment.

  2. The framework relies heavily on calling pre-existing external models (e.g., ResNet50 and RandomForest) inside the sandbox, while the proposed AL-GRPO yields only marginal improvements over standard GRPO. Moreover, the evaluation setup is unfair to end-to-end visual table baselines that are only shown table headers without full table images.

Questions for authors
  1. The introduction highlights visual parsing challenges such as degraded scans and complex table layouts, yet the sandbox already provides clean and complete CSV files while the visual input only shows a header preview. Please clarify the necessity of the visual modality in Table QA when the full structured data is directly accessible via code.

  2. In Table 1, end-to-end visual models like Table-LLaVA and HIPPO get very low scores (e.g., 11.75% and 7.86% on WikiTQ) because they only see table headers rather than full table images, and they are not trained as coding agents. Additionally, in Table 4 and Appendix E.2, TWT trains a RandomForest and calls a pre-trained ResNet50Predict() in the sandbox, combining their outputs with heuristic weights (e.g., 0.1 and 0.9). It would be helpful to discuss how to ensure a fair comparison for end-to-end visual baselines, and how TWT learns genuine multimodal feature interactions beyond heuristically weighting two external models.

  3. AL-GRPO mainly masks out failed code segments during loss computation, and its improvement over standard GRPO in Table 2 is relatively modest (e.g., 78.39% to 78.95% on WikiTQ, and 0.101 to 0.093 nRMSE on Paintings). Please provide more insights or statistical evidence on the effectiveness of AL-GRPO compared to standard GRPO.

3 comments

We thank the insightful feedback of reviewers.

W1 & Q1

Regarding the visual modality, having access to data and correctly extracting and using those data are two different things. With the same resources, the base Qwen3-VL-8B achieves 50.64% accuracy on WikiTQ, while TWT achieves 78.95%, indicating that this interactive task still involves substantial challenges in information acquisition and reasoning.

In table QA, the incomplete table helps the model understand the table format, thereby providing better guidance for subsequent code-based access. In prediction tasks, the visual modality analyzes relevant image information: images are not merely table-header previews but also provide information about the instance to be predicted. After removing explicit tabular features in Table 3, TWT still achieves 31.40% on Adoption, close to 32.90% with complete features. The visual modality is an indispensable part of this process and is therefore a necessary component.

W2 & Q2 & Q3

We first clarify a fact: these baselines are not restricted to accessing table headers. All baselines receive the same sandbox instructions and resource access, so complete tables are not available exclusively to TWT. Table 1 compares different models under the same interactive task interface, including their ability to acquire information through code.

Notably, even when complete tables are provided, Table-LLaVA achieves 16.62 on WTQ and 53.20 on TabMWP, while HIPPO achieves 85.83 on WTQ and 49.10 on TabMWP. Together with the results in “Comparison to Tabular/Image Baselines,” TWT remains the best overall on this class of tasks, supporting the paper’s conclusions. The overall performance advantage cannot be attributed to selecting visual table baselines that are poorly adapted to coding interfaces.

Regarding code agents, proprietary models possess this capability, and the Codex experiment also does not outperform TWT. The comparison is fair: different methods employ training approaches tailored to different problems. The poor performance of comparison methods on TVMU highlights the importance of this problem rather than an unfair comparison. The experimental results demonstrate the effectiveness of our method.

Regarding multimodal interaction, TWT dynamically fuses predictions based on data quality, which differs from fusion approaches such as fixed weighting. We present the weighted-fusion results on Paintings alongside our results below. The experiments demonstrate that our method outperforms weighted fusion. We will add further analysis to explain what kind of modality fusion TWT has learned.

Tabular weight wPaintings nRMSE ↓
0 (image only)0.1521
0.10.1477
0.20.1437
0.30.1403
0.40.1373
0.50.1348
0.60.1329
0.70.1316
0.80.1309
0.90.1308
1 (tabular only)0.1313
TWT0.093

Regarding GRPO, citing only the 0.56-percentage-point difference on WikiTQ does not fully reflect the results in Table 2. Compared with GRPO, AL-GRPO improves TAT-QA accuracy by 1.45 percentage points and SkinCA accuracy by 2.73 percentage points, while reducing Paintings nRMSE by approximately 7.9% relatively. Code execution success and problem-solving convergence rates also improve further. More importantly, standard GRPO performs slightly below SFT on both prediction tasks, whereas AL-GRPO outperforms SFT on both. This indicates that the gains do not simply come from “adding another round of RL training.”

The design motivation also directly corresponds to these interactive tasks: a correct final answer may result from recovery after failed calls, and therefore should not lead to the same positive advantage being indiscriminately assigned to all preceding code segments. AL-GRPO uses execution status to select code segments’ contributions to the policy loss, directly addressing this issue.

Reviewer_eXBaOpenReview ↗

Reply

Thanks for addressing my questions. I did find some clarifications helpful, yet the fundamental limitations regarding assumptions on access to meta-data still persist, and will ultimately impact this work's positioning and real-world practicality.

Thank you for the follow-up. We respectfully disagree that the assumption about metadata access substantially limits the practical relevance of this work. In financial analysis, an analyst may begin with only filing metadata, a table header, or a few visible rows. Identifying which additional records to retrieve is itself part of the task.

Consider an analyst asked how a company’s operating cash flow changed between two fiscal years. An entry in the SEC’s EDGAR structured disclosure feed provides the company name, filing date, form type, and a link to the filing. From that starting point, the analyst must locate the statement of cash flows, identify the operating cash flow line, and retrieve the figures for the two years. A financial database offers a similar example at the table level. In BigQuery, an analyst can inspect a table’s schema or preview its first few rows. Even if the relevant company and years fall outside that preview, the column names provide enough structure to write a targeted query.

TVMU captures precisely this aspect of the workflow. The model initially sees the table header and must decide what evidence to retrieve through code. Table 1 gives every method the same initial view, sandbox instructions, and access to the underlying table. Performance therefore reflects, in part, how effectively each method can turn a partial view into an accurate answer. We will make this practical motivation explicit in the revision. Given these concrete use cases and TWT’s performance under shared access conditions, we would be grateful if you would consider raising your score.

Reviewer_grhmOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

TWT trains a multimodal model to answer questions and make predictions by alternating between reasoning, Python execution, and inspection of the results. The model can access raw tables, images, and pretrained models in a sandbox, allowing it to query data, train tabular predictors, and combine their outputs with image predictions. Training begins with supervised fine-tuning on demonstrations checked against the target answers, followed by reinforcement learning using final-answer rewards. The proposed AL-GRPO uses execution feedback to exclude failed code segments from the policy loss. The authors report improvements across table question answering and prediction tasks, with further analyses of the training stages, code execution success, and performance when tabular features are absent from the visual input.

Critical strengths
  • Using Python to read tables, perform calculations, and fit predictors is a sensible choice for these tasks. The same agent can handle both question answering and prediction, and the examples show how it works in practice.

  • The trained model improves substantially over its starting point on both question answering and prediction, and is competitive with larger models on several benchmarks.

Critical weaknesses
  • The main new component, AL-GRPO, is not described clearly enough to reproduce. The paper says failed code segments receive no gradient, but does not explain whether they remain in the context for later steps or how the remaining loss is normalized.

  • Most of the improvement comes from supervised fine-tuning. AL-GRPO improves on this starting point, but the paper does not report variation across runs. For prediction, the supplied tabular and image models are already strong; a simple fusion of the same predictors would help show what the agent loop adds.

Questions for authors

Which tokens does AL-GRPO mask, and how are conditioning history and loss normalization handled?

1 comment
AuthorsOpenReview ↗

Author Response

We thank the reviewer for valuable feedback.

W1&Q1

During the GRPO stage, all code segments that encounter errors (as reported by the sandbox environment) are excluded from loss computation. We use the loss-scale feature of ms-swift for context management. This design addresses a specific problem: when the model corrects an error after a failed call and eventually answers correctly, a standard trajectory-level advantage may reinforce both the failed call and the effective correction steps. AL-GRPO excludes the direct policy-loss contributions of failed code, so that failed actions in successful trajectories do not receive the same positive reinforcement solely because the final answer is correct.

W2

SFT providing the main initial improvement does not contradict AL-GRPO providing an additional contribution. The two stages serve different functions: TO-SFT establishes basic task procedures and tool-use capabilities, while AL-GRPO optimizes interactive decisions after these capabilities have been developed.

MetricTO-SFT+GRPO+AL-GRPOAL-GRPO relative to GRPO
WikiTQ ACC78.3878.3978.95+0.56 percentage points
TAT-QA ACC71.3171.3772.82+1.45 percentage points
SkinCA ACC71.0870.9273.65+2.73 percentage points
Paintings nRMSE0.0990.1010.093Approximately 7.9% reduction

The key finding is not only that AL-GRPO outperforms SFT, but also that standard GRPO fails to improve upon SFT on SkinCA and Paintings, whereas AL-GRPO achieves better results on all four evaluated tasks. Therefore, its contribution cannot be summarized solely by the smaller gain on WikiTQ.

Furthermore, Figure 4 shows that, compared with SFT, the complete TWT system reduces the non-convergence rate from 2.6554% to 0.8339% on Adoption without tabular features, and from 6.7942% to 0.5375% on Paintings, corresponding to relative reductions of approximately 68.6% and 92.1%, respectively. These results indicate that the improvements after training are also reflected in the ability to complete interactions.

Regarding modality fusion, we conduct modality fusion experiments using different weights. In the table, w denotes the tabular modality weight, and the corresponding image modality weight is 1-w.

Tabular weight wSkinCA ACC (%) ↑Paintings nRMSE ↓
0 (image only)68.910.1521
0.169.340.1477
0.269.780.1437
0.370.650.1403
0.472.390.1373
0.573.910.1348
0.676.080.1329
0.778.690.1316
0.878.260.1309
0.976.740.1308
1 (tabular only)74.780.1313
TWT73.650.093

TWT can effectively use different resources within a unified framework and remain competitive across different tasks. The experiments show that TWT at least performs better than the results of one modality. Since the ensemble weights and the importance of each modality are unknown when encountering a new task, TWT's reasoning approach of dynamically ensembling modalities by analyzing data quality has practical application value.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for your time and constructive feedback. We would like to follow up on our point-by-point responses and the revised manuscript, which includes additional experiments and clarifications addressing the concerns raised in your reviews.

We would be grateful if you could review these updates and let us know whether they address your main concerns. If any issues remain, we would welcome the opportunity to discuss them and provide further clarification.

We look forward to hearing your thoughts and continuing the discussion.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.