- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PRISM. It is a budget aware framework for TableQA. It jointly selects an LLM and table representation, with the intention of lightweight model for initial inference, verification, then escalation in uncertain responses to stronger models. Experiments on HiTab and WTQ achieve 83% and 78.04% accuracy, and report improved cost-normalized performance, but with a tradeoff in accuracy and latency.
Critical strengths
- Tackles a practical problem with a contribution for measurable efficiency. It addresses the balance between accuracy, cost, and latency with metric per $ than raw score.
- The results have confidence intervals, ablations, and cost accounting. PRISM supports argument that routing design works.
- Easy to follow framework for reproducibility.
Critical weaknesses
- Limited isolation of the routing when mixed with representation. For example. HiTab seems to use only CSV for primary routing, but the fallback changes the model capacity and representation.
- Generalization claims: The acceptance mechanism relies on self-reported confidence. How are the authors calibrating this measure, and what are the false positives, false negatives here? Furthermore, how much does confidence thresholds contribute to the routing?
- Single model family: PRISM is evaluated on a single model family and two benchmarks. Does this port to other models, and the decisions made during tuning your prompts, parameters, those this work on OOD table benchmark?
Questions for authors
- Clarify how much PRISM's improvement is due to representation selection vs model escalation?
- Generalization results for OOD table QA not used to calibrate framework?