- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper provides a systematic analysis of the tabular-vision multi-modal understanding task and propose Thinking with Tables, a framework that utilizes a code-based interactive reasoning mechanism to model the complex tabular information.
Critical strengths
- The paper proposes a program-aided, interactive reasoning mechanism that bridges multimodal inputs, successfully handling structural variability and data incompleteness.
- The proposed TWT combines TO-SFT to internalize diverse task pipelines with AL-GRPO to optimize execution stability and reasoning convergence.
- TWT consistently outperforms existing open-source baselines across diverse tasks and datasets.
Critical weaknesses
- TWT is a method designed for the TVMU task which has a high degree of domain specificity. TWT is not evaluated on broad foundational benchmarks, leaving its generalizability beyond specialized tabular-visual domains unverified.
- The training pipeline depends heavily on synthesizing reasoning traces using advanced proprietary or larger models, which introduces teacher bias and limits self-contained scalability.
Questions for authors
- The paper brands TWT as a neuro-symbolic reasoning framework, yet the symbolic component is essentially standard Python code generation executed in a sandbox via code interpreters. How does TWT genuinely incorporate symbolic reasoning beyond standard tool-aided code execution? Is the term "neuro-symbolic" overstated for what is fundamentally a code-interpreter agent loop?
- Given that TWT is specifically trained and optimized on synthesized domain-specific trajectories (via TO-SFT and AL-GRPO), to what extent is the performance comparison against zero-shot or general-purpose MLLMs entirely fair? Would TWT retain its performance advantage if general baseline models were also given access to equivalent in-domain supervised fine-tuning?
- The framework relies heavily on iterative interactions with a Python sandbox environment, but what happens when the model generates syntactically correct code that contains semantic logic errors or hallucinations regarding tabular data structures?
- The paper lacks an evaluation of computational efficiency (latency, multi-turn sandbox overhead, and token cost) as well as broader generalization benchmarks. Could the authors provide a comprehensive analysis of end-to-end inference latency under multi-turn code execution and report performance on general multimodal benchmarks?
4 comments
Contd.
Q1
Symbolic reasoning does not necessarily require a theorem prover or a dedicated logical language. In the era of large language models, reasoning is mainly enhanced through three symbolic paradigms. Our work follows the program-aided paradigm: the neural model translates natural-language intent into explicit variables, conditions, filtering, and computational operations, which are then executed by an interpreter according to program semantics.
Compared with generating answers solely in natural language, this process converts part of the reasoning into inspectable and executable structured operations. The contribution of this paper is to address TVMU through a neuro-symbolic (NeSy) system that unifies table question answering and prediction procedures and achieves strong performance.
Reference: Xiao-Wen Yang, Jie-Jing Shao, Lan-Zhe Guo, Bo-Wen Zhang, Zhi Zhou, Lin-Han Jia, Wang-Zhou Dai, and Yu-Feng Li. Neuro-symbolic artificial intelligence: Towards improving the reasoning abilities of large language models. IJCAI Survey.
Q2
First, the general MLLMs receive the same resources, making the comparison fair in terms of the evaluation protocol. TWT’s ability to outperform models with more parameters itself demonstrates the effectiveness of our proposed fine-tuning method.
Second, regarding tabular models, they have all been trained using table-related domain knowledge, have their own training methods for solving tabular problems, and may even use larger amounts of training data.
To further illustrate this point, taking table QA as an example, we evaluate these models following their training and testing settings. Some tabular models, even when provided with complete tables, perform worse than TWT without complete tables in its input: Table-LLaVA achieves 16.62 on WTQ and 53.20 on TabMWP, while HIPPO achieves 85.83 on WTQ and 49.10 on TabMWP. Together with the results in “Comparison to Tabular/Image Baselines,” TWT remains the best overall on this class of tasks.
We fine-tune InternVL3.5-8B and obtain the following results:
| Method | TabMWP ↑ | WikiTQ ↑ | FinQA ↑ | TAT-QA ↑ | Adoption ↑ | SkinCA ↑ | Pawpularity ↓ | Paintings ↓ |
|---|---|---|---|---|---|---|---|---|
| TWT (Ours) | 97.10 | 78.95 | 61.95 | 72.82 | 32.90 | 73.65 | 0.199 | 0.093 |
| InternVL3.8-8B | 95.10 | 76.85 | 60.95 | 71.28 | 31.50 | 69.78 | 0.0206 | 0.102 |
TWT achieves better performance overall. Meanwhile, the performance improvements of the InternVL model also provide strong evidence for the effectiveness of our training pipeline.
Q3
TWT performs subsequent reasoning based on code execution outputs. When hallucinations occur, particularly hallucinations about table structures, the returned code execution results may differ from the model’s expectations. Subsequent reasoning therefore provides an opportunity to correct these errors.
Q4
Regarding computational efficiency, we separately measure latency, sandbox overhead, and token usage for QA and prediction tasks.
| Type | Latency | Sandbox overhead | Tokens | LLM calls |
|---|---|---|---|---|
| Table QA | 26.08 s | 2.17 s | 1239 | 3.22 |
| Table Prediction | 102.54 s | 4.46 s | 37863 | 7.06 |
For general multimodal benchmarking, results are shown in W1. The experimental results show that TWT not only achieves superior performance on tabular tasks but also exhibits strong general capabilities.
Thanks for addressing my questions. However, I still have reservations about the theoretical innovation and practical value of the method, so I maintain my score.
Thank you for your follow-up and for the time and effort you have devoted to reviewing our work. We appreciate your consideration of our responses. To better understand your remaining reservations about “theoretical innovation” and “practical value,” we would appreciate clarification on which specific concerns remain unresolved and why our explanations have not sufficiently addressed them.
Regarding innovation, your original question concerned the substance of the neuro-symbolic characterization relative to standard code-interpreter agents. If this concern remains, could you please identify which claimed distinction our response failed to establish? If “theoretical innovation” refers to an additional concern beyond the original question, it would be helpful to understand which specific contribution you find insufficient and the reasoning behind that assessment, so that we can respond meaningfully.
Regarding practical value, your original review kindly acknowledged TWT’s ability to handle structural variability and data incompleteness, together with its consistent improvements over open-source baselines across diverse tasks and datasets. We believe these strengths provide concrete evidence of utility within the intended application setting. Could you please clarify how these acknowledged benefits are weighed against your remaining concerns about generalization, scalability, reliability, or efficiency, and which aspects of our responses you find insufficient?
We would therefore be grateful if you could reconsider the assessment in light of our responses and clarify any remaining grounds for the current score. Connecting the broader reservations to specific unresolved issues would help us better understand your assessment and allow for a more constructive discussion of the paper’s contributions and limitations. Thank you again for your time and engagement.
We thank the reviewers for the valuable review.
W1
This paper proposes a task framework for tabular understanding. The same framework and interaction protocol cover four question-answering datasets and four prediction datasets spanning classification and regression. The differences across question answering, classification, and regression constitute precisely the heterogeneity of task procedures that this paper aims to address, rather than designing a solver for a single dataset.
Generalization ability is a valuable question discussed in our conclusion. For general multimodal benchmarking, we additionally conduct experiments on the MMMU dataset and compare TWT with tabular-related open-source MLLMs. The results are as follows:
The experimental results show that TWT not only achieves superior performance on tabular tasks but also exhibits strong general capabilities.
W2
Training does not stop at imitating teacher trajectories. After TO-SFT, AL-GRPO constructs rewards using the final task answers and continues optimizing the model. Table 2 shows the additional gains of this stage over SFT. In Table 1, the 8B TWT model also outperforms Qwen3-VL-Plus on six of the eight tasks. These results indicate that the final system’s task performance is not simply equivalent to directly invoking the teacher model.
The performance gains of our method come from supervised trajectories filtered by their outcomes establishing initial capabilities, followed by task rewards and environment interactions further optimizing the policy. Teacher bias may remain in the initial data, which is a limitation of the data-generation method that we will discuss further.