acceptodds
Withdrawn from ICLR 2027

Is LLM-Based Tabular Analysis Invariant to Linguistic Realization?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language models (LLMs) increasingly analyze tabular data through textual representations, allowing them to exploit semantic cues from column names, categorical values, and other natural-language elements. This capability, however, raises a fundamental question: whether LLMs can access and manipulate the same structured information consistently when its linguistic realization changes. We study this problem through a controlled English–Korean setting in which every query is fixed in English and semantically aligned tables are presented in either English or Korean. Our evaluation framework combines paired synthetic and real-world tables, diverse tabular analysis queries, multiple serialization formats, and paired metrics that distinguish aggregate performance differences from instance-level inconsistency. Across multiple LLM families, we find systematic but highly non-uniform sensitivity to linguistic mismatch. English tables generally yield higher accuracy, yet aggregate English–Korean performance gaps can substantially understate behavioral instability: models frequently change correctness across paired instances even when their average accuracy difference is small. Higher overall performance and larger models also do not consistently imply greater cross-lingual stability. Sensitivity varies sharply across query types and is most pronounced in cross-lingual linking, while remaining substantial even for basic column grounding. These results show that aggregate accuracy alone is insufficient to characterize whether LLMs consistently access semantically equivalent tabular information, highlighting the importance of paired cross-lingual evaluation for LLM-based tabular analysis.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
table understanding, table representation, tabular reasoning, serialization, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wbrpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on the robustness analysis of LLMs’ table analysis in the English-Korean multilingual scenario, and demonstrates the characteristics of LLMs in analyzing tables across these two languages through controlled analysis.

Critical strengths

The overall writing of the paper is relatively clear. The scenario this paper focuses on--LLM’s table understanding in multilingual settings -- is both valuable and interesting.

Critical weaknesses

1.Linguistic Realization is a big concept. Determining whether LLMs are robust in analyzing tables under this effect requires extensive experiments (even beyond simple variations in language types). However, the author's focus is limited to different types of languages, even just the English-Korean setting. Its relatively limited results simply cannot respond to the grand claim in the title.

2.Even when focusing on the specific scenario of English-Korean cross-lingual table analysis, the authors' analysis is insufficient, and the claims made are trivial. The overall research quality is significantly lower than previous works.

Questions for authors
  1. Compared to previous works focusing on table analysis of LLMs in multilingual scenarios (such as MultiTAT[1] and M3TQA[2]), what further achievements does this work report?

[1] Zhang, Xuanliang, et al. "MULTITAT: Benchmarking Multilingual Table-and-Text Question Answering." EMNLP (Findings). 2025. [2] Shu, Daixin, et al. "M3TQA: Massively Multilingual Multitask Table Question Answering." Findings of the Association for Computational Linguistics: ACL (Findings). 2026.

Reviewer_sb9GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines robustness to linguistic realization in LLM-based tabular analysis. The authors fix the questions in English and present semantically aligned tables in either English or Korean, thereby comparing a language-matched English-table condition with a cross-lingual condition in which English questions access Korean tables. The paper constructs a paired English–Korean benchmark spanning synthetic and real-world data, seven table operations, three table serializations, and six LLMs. Beyond type-macro accuracy, it uses the Correctness Flip Rate (CFR) to measure whether correctness changes for the same semantic instance across the two table-language conditions. The authors report that the Korean-table condition has lower aggregate accuracy for every evaluated model, while small English–Korean average accuracy gaps can still coexist with substantial instance-level correctness flips.

Critical strengths

S1: The CC/CW/WC/WW outcomes and CFR in Section 3.2 directly address whether behavior changes for the same instance. The coexistence of small gaps and high CFRs in Table 2 effectively illustrates that aggregate accuracy can mask instance-level instability.

S2: The experiments cover multiple relevant factors. The study spans seven operations, synthetic and real-world data, multiple model families and scales, and Markdown, JSON, and KV serializations. This coverage enables the paper to show that sensitivity is not uniform across tasks, models, or representations.

Critical weaknesses

W1: CFR is the paper’s central metric, but the paper does not adequately specify how cross-lingual answers are normalized and scored. In particular, for entities such as book titles and person names that may have English, official Korean, or transliterated forms—as well as multi-entity and numeric answers—unspecified matching rules could directly affect CW, WC, and CFR. The authors should provide answer-normalization rules, scoring scripts, and instance-level labels to verify that the reported flip rates reflect model behavior rather than evaluation artifacts.

W2: The one-directional language design does not isolate query–table mismatch from Korean-language competence. The study compares English queries over English tables with the same English queries over Korean tables. Therefore, the observed degradation and correctness flips may arise from query–table language mismatch, lower Korean-language table understanding, or their combination. The current evidence supports a directional finding—performance differs when fixed English queries access Korean rather than English tables—but does not by itself establish a general causal effect of linguistic mismatch. A balanced query-language × table-language control, ideally including Korean-query/Korean-table and Korean-query/English-table conditions, would distinguish mismatch from language-specific competence. If a full evaluation is infeasible, this analysis should be conducted on a representative subset, or the causal claims should be narrowed accordingly.

Questions for authors

Q1: Please specify exactly how correctness is determined under English and Korean table conditions.

Q2: Have the authors run balanced conditions such as Korean-query/Korean-table and Korean-query/English-table?

Q3: Could the authors report the English–Korean token-length distributions under each evaluated model’s tokenizer, together with the handling of inputs near the context limit? Since semantically aligned English and Korean tables may have systematically different token lengths and tokenization patterns, it would be useful to clarify whether the observed gaps and flips persist after controlling for input length or excluding long-table cases.

Reviewer_5wzNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether LLMs answer table questions consistently when the same table is written in English vs Korean, with the question always in English. It builds 306 paired tables (300 synthetic, 30 rows x 6-8 columns from bilingual vocabularies; 6 real-world tables from books, FIFA players, cars and Texas real estate, manually translated) and 1,520 table-question pairs over 7 query types (column grounding, row selection, aggregation, arithmetic, comparison, multi-step, cross-lingual linking), each run in 3 serializations (Markdown, JSON, KV), for 9,120 instances. Metrics are type-macro accuracy, the EN-KO gap, and a Correctness Flip Rate (CFR = share of pairs that are right in one language and wrong in the other). six models are tested zero-shot (GPT-4o-mini, GPT-5.4-mini, Llama-3.1-8B, Llama-3.2-3B, Gemma-3-12B/27B). Korean tables lower accuracy for all models (gap 3.1-10.9 pts), CFR (13-23%) is often much larger than the gap, bigger/stronger models are not always more stable, cross-lingual linking is the most affected type, the book subsets show larger gaps as title translation gets freer, and no serialization removes the gap.

Critical strengths
  1. The paired design is clean and the right way to ask this question. Keeping rows, values, structure and the English query fixed and changing only the table language isolates the effect better than prior multilingual table benchmarks (MultiTAT, TableEval) that compare aggregate scores across different data. The book subsets (transliteration / literal / free translation of official titles) are a nice natural control, and the gap and CFR grow in the expected order (8.9 -> 10.7 -> 17.1 pts gap).

  2. The point that an aggregate gap can hide instance-level instability is useful and clearly shown (e.g. Llama-3.1-8B: 3.1 pt gap but 18.7% CFR; cross-lingual linking for Llama-3.1-8B: 0.48 pt gap but 41% CFR). Reporting the full CC/CW/WC/WW breakdown is a good practice that other multilingual evaluations could adopt.

Critical weaknesses
  1. CFR has no noise floor, so the main claim is hard to interpret. Accuracies are low (EN 17-46%, real estate ~12%), and the paper does not say what temperature/decoding was used or whether runs were repeated. At these accuracy levels, a sizeable share of flips may come from ordinary instability that has nothing to do with language. The design already allows a simple baseline: the flip rate between two serializations in the same language (EN-Markdown vs EN-JSON), or between two runs of the same EN instance. Without this, "CFR 18.7% despite a 3.1 pt gap" could mostly be noise, and claims like "larger models are not more stable" (based on two model pairs, where Llama-3.2-3B vs 3.1-8B also differ in version) are not supported. No confidence intervals or paired tests (e.g. McNemar) are reported for any gap.

  2. Answer scoring is not described, and it can drive the whole effect. The paper says "language-appropriate answer forms are accepted" but does not say how answers are matched (exact match, normalization, LLM judge?) or what happens when the model answers in English for a Korean table (e.g. "Hermann Hesse" vs "헤르만 헤세"), or uses a different romanization. Very low absolute accuracy on simple lookups ("Who wrote Demian?") also hints at strict matching. If the KO condition is penalized partly because of answer-language mismatch, the gap measures output formatting rather than "access to structured information". Related: with the query always in English, the design cannot separate "Korean table" from "language mismatch" (no KO-query/KO-table condition), and the cross-lingual linking type only requires cross-lingual linking in the KO condition, so its large gap is partly by construction. The scope (one language pair, 6 real tables, small/older models, no reasoning or CoT, Qwen dropped) is also narrow for the general claim in the title.

Questions for authors

Questions that could change my assessment:

  1. How are answers scored? Exact match, normalization, alias lists, or a judge? In the KO condition, is an English answer to a Korean table (or a different romanization) marked correct? What share of KO errors are answer-language/format errors vs truly wrong values?
  2. What is the flip rate within the same language, e.g. EN-Markdown vs EN-JSON, or two repeated EN runs? Please report it per model next to CFR so we can see how much of CFR is due to language.
  3. What decoding settings were used (temperature, sampling, max tokens), and were results averaged over seeds? Can you add CIs / McNemar tests for the gaps in Tables 2-6?
  4. Why are accuracies so low on simple lookups (e.g. real estate ~12% in English)? A short error analysis of EN failures would help show the benchmark is measuring what you intend.
  5. Did you consider a KO-query + KO-table condition? Without it, how do you separate the effect of "Korean table" from "query-table mismatch"?

Minor / clarifications: App A "Book. The automobile data are derived from the May 2026 Library ..." looks like a copy error; book data are originally Korean while the other domains are translated from English, which may affect the domain comparison; section numbering "3.0.1"; Figure 1 "Evalution"; Figure 3 fonts are not embedded properly (text is garbled when extracted); Table 3 caption mentions bold values but none are visible.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.