acceptodds
Withdrawn from ICLR 2027

PRISM: Pareto Routing over Input Structure and Models for Budgeted Table Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Table question answering (Table QA) requires a model to derive an answer from a variably-structured table and a natural-language question. Although general-purpose LLMs, including GPT-class systems, and smaller language models (SLMs) can solve many such questions, a fixed model and fixed serialization force a deployment to choose between accuracy, cost, and latency before the difficulty of each input is known. Thus, we introduce PRISM, Pareto Routing over Input Structure and Models, which treats the model–representation pair as the primary decision and uses a second, evidence-gated call only when the first response fails a prespecified acceptance screen. The screen combines answer availability, evidence consistency, and reported confidence; a deterministic quota bounds mean API use by 1.25 calls per question. We derive a risk-priced continuation boundary, conditional representation–model crossover guarantees, and a risk bound that accounts for denied recovery. Completed experiments reach 83.02% strict accuracy on HiTab and 78.04% denotation accuracy on WikiTableQuestions (WTQ). On 851 paired WTQ questions, PRISM yields 43.43% more correct answers per accounted dollar than Pro/CSV, while trading away 8.96 accuracy points and adding 2.99 seconds of logical latency. The results support selective recovery and make its resource trade-offs explicit; they do not claim calibrated uncertainty or universal Pareto dominance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Table Question Answering, LLM Routing, Model Cascades, Input Representation, Selective Prediction, Budgeted Inference, Pareto Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_JX9GOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PRISM. It is a budget aware framework for TableQA. It jointly selects an LLM and table representation, with the intention of lightweight model for initial inference, verification, then escalation in uncertain responses to stronger models. Experiments on HiTab and WTQ achieve 83% and 78.04% accuracy, and report improved cost-normalized performance, but with a tradeoff in accuracy and latency.

Critical strengths
  1. Tackles a practical problem with a contribution for measurable efficiency. It addresses the balance between accuracy, cost, and latency with metric per $ than raw score.
  2. The results have confidence intervals, ablations, and cost accounting. PRISM supports argument that routing design works.
  3. Easy to follow framework for reproducibility.
Critical weaknesses
  1. Limited isolation of the routing when mixed with representation. For example. HiTab seems to use only CSV for primary routing, but the fallback changes the model capacity and representation.
  2. Generalization claims: The acceptance mechanism relies on self-reported confidence. How are the authors calibrating this measure, and what are the false positives, false negatives here? Furthermore, how much does confidence thresholds contribute to the routing?
  3. Single model family: PRISM is evaluated on a single model family and two benchmarks. Does this port to other models, and the decisions made during tuning your prompts, parameters, those this work on OOD table benchmark?
Questions for authors
  1. Clarify how much PRISM's improvement is due to representation selection vs model escalation?
  2. Generalization results for OOD table QA not used to calibrate framework?
Reviewer_NvTmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper focuses on selecting an appropriate combination of LLM and table representation under limited inference budgets. The authors consider larger models that generally provide better performance and smaller models that are more cost-efficient, along with different table representations such as HTML, Markdown, and CSV. The model and representation pair is selected based on a routing score that considers the estimated performance of each pair for a given query, inference cost, and manually defined preferences reflecting which combinations are expected to perform well. After selecting the pair, the LLM generates an answer and provides a confidence score estimating its correctness. Based on this confidence score and evidence verification, the method decides whether to accept the answer or invoke a fallback model. The proposed method shows performance improvements on WikiTQ and HiTab.

Critical strengths
  • The paper addresses the trade-off between performance and inference cost, which has been an important topic in recent LLM reasoning research, and applies it to Table QA.

  • Previous studies have explored routing among different table representations, but this paper considers jointly selecting the model and table representation while accounting for inference efficiency. This is a meaningful extension of existing table representation routing methods.

Critical weaknesses
  • The method for determining the routing score is unclear. The paper calculates a routing score for each model and representation pair based on the context bucket (b(x)b(x)). However, it is unclear how well this approach generalizes to other settings, as the paper uses different methods for HiTab and WikiTQ. For HiTab, the estimated performance of each pair is derived from development data, whereas WikiTQ relies on synthetic priors and manually defined preferences. Furthermore, the paper provides limited analysis of how well the estimated accuracy (p~a(x)\widetilde{p}_a(x)) matches the actual accuracy of each model and representation pair. Since these estimates directly affect the routing decisions, a more detailed analysis of their reliability would be helpful.

  • The method for assigning queries to context buckets relies on simple heuristics. The paper classifies queries into arithmetic and lookup categories based on predefined keywords. However, the required operation is not always explicitly stated in the question, so this approach may not reliably identify the correct query type. It would be helpful to provide separate performance analyses for arithmetic and lookup queries to better understand how this classification affects routing performance.

  • The paper states that the LLM is prompted to provide a confidence score representing the estimated probability that the entire answer is correct. However, the exact prompt used to generate this score is not provided. Including the prompt would help clarify how the confidence scores are obtained.

  • The experiments are limited to WikiTQ and HiTab, both of which contain tables collected from Wikipedia. This leaves some questions about how well the proposed method generalizes to other domains. Additional experiments on datasets such as FinQA would be helpful. The method is also not evaluated on large-scale database benchmarks such as BIRD and BEAVER, or in settings involving multiple tables, such as MMQA.

Questions for authors
  • For HiTab, the paper uses development data to estimate the performance of each model and representation pair, while for WikiTQ, it relies on synthetic priors and manually defined preferences. Could the authors explain why different approaches were used for these two datasets?

  • How sensitive is the routing performance to the composition of the development data? For example, how would the performance change if the development data came from a different domain? It would also be helpful to understand how the size of the development data or an imbalanced distribution of question types affects routing performance.

Reviewer_bx2zOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PRISM, a budgeted cascade for table question answering. A selector picks a primary model and table serialization. A verifier checks the response for answer availability, confidence, and consistency with cited cells. Responses that fail request one escalation to a stronger model under a fixed quota. The paper derives a continuation boundary, a representation–model crossover condition, and a risk bound. On HiTab and WikiTableQuestions with three Gemini models, PRISM improves over its own primary and over a same-model retry. Against always calling the strongest model, it trades about nine points of accuracy for lower cost.

Critical strengths
  1. The experimental accounting is rigorous. Every attempt, failure, rescue, and harm is logged, including calls whose cost the API did not report. The shadow continuation in Section 6.4 compares escalation against a same-model retry on identical primaries. This is consequential because it makes the recovery result credible.

  2. The representation comparison is properly controlled. The serializers are verified by round trip to carry identical content (Definition 1), so format effects cannot come from added or missing information.

Critical weaknesses
  1. The evidence does not support joint model–representation routing, the paper's title contribution. HiTab uses only CSV primaries. The WTQ selector runs on hand-set preferences. The format comparisons are not significant (Appendices C.2 and F.2). Most of the gain comes from escalation repairing missing or truncated answers, many caused by the output-token cap on Flash (Section 6.1). The authors ought to separate model from format in the escalation. If representation adds nothing measurable, the claims should be scoped to selective recovery.

  2. PRISM is compared only against its own variants and a single strong model (Pro/CSV). Pro/CSV is more accurate and faster. Cheaper policies return more correct answers per dollar (Table 20). No existing cascade is evaluated, and the closest prior work is not cited. Yue et al. (ICLR 2024) escalate when answers disagree across reasoning representations. AutoMix (Aggarwal et al., NeurIPS 2024) escalates on self-verification under a budget. Liu et al. (NAACL 2024) study same-content table variants on WTQ. The authors ought to position against these works and compare against a cost-matched cascade.

Questions for authors
  1. How much of the WTQ gain over the primary remains when the primary runs with a larger output-token limit?

  2. What does escalation to Pro/CSV give under the same gate? This separates model from format in the continuation.

  3. How does PRISM compare with a cost-matched existing cascade, such as AutoMix, on the 851-question cohort?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.