acceptodds
Withdrawn from ICLR 2027

How Deep Is Your Value? Table-Cell Operands Let Go of VLM outputs Earlier Than Answers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) answer questions about tables directly from document page images, yet final-answer accuracy does not reveal how the visual evidence of a cell is processed inside the model. This matters because a cell’s value can serve two roles: in LOOKUP, it is read out as the answer; in COMPUTE, it is used as an operand with values from other cells, often on other pages. A model could process every cell to the same depth regardless of its role, or release an operand once it has been used. We construct paired LOOKUP and COMPUTE instances that share a table, a target cell, and a pixel edit to that cell. In eight VLMs, we first verify that the cell’s visual tokens carry the value used to construct the answer, and then intervene on these tokens layer by layer. We find that operands let go of the output earlier than answers: replacing the cell’s value redirects the output only up to an earlier layer in COMPUTE than in LOOKUP, in all eight models under a log-probability criterion and in six under full generation. The difference lies in the value, not in the cell’s presence: the depth at which removing the cell stops mattering is set largely by the model and barely moves across question types or page structures. An operand’s value remains linearly decodable at the cell after its influence ends, but later text positions carry the computed result rather than an operand the model can recompute from. Code used for the analysis is publicly available at https://anonymous.4open.science/r/vlm-table-cell-layer-analysis-67D8/.

Primary area:
interpretability and explainable AI
Keywords:
Document Table VQA, Vision-Language Models, Mechanistic Interpretability, Visual Representations, Layer-wise Analysis

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wB7ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper is a mechanistic examination of how VLMs perform table visual question answering. Specifically, it traces how the relevant cells’ information are processed within the model by quantifying the information’s ``intervention endpoint”, which is defined as the last layer at which patching in an alternative hidden state will change the model’s output. The paper shows that the effective depth varies based on the role of the cell value: when a value is read out in the answer (a LOOKUP operation), it has a deeper effective depth than when the value is used as an operand (a COMPUTE operation).

Critical strengths
  • The experimental design is sound: the taxonomy of “seen” vs. “used” cells is a useful one, and the method of validating the intervention site and thresholds are thoughtful.
  • The experimental results are thorough: The paper’s conclusions are drawn from experiments on 8 different models. Additionally, confidence intervals are provided for substantive claims as in Table 1.
Critical weaknesses
  • I’m not sure what the practical value of this paper has. It presents an observation about the relative intervention “depth” of information, but is unclear what the utility of this observation is. How can this observation help us improve future models, or how we use current ones?
  • The paper’s presentation, specifically the writing, needs substantial improvement. Specifically, the setup sections would benefit from editing passes to improve style and clarity.
Questions for authors
  1. What are the practical implications of the findings in the paper? How does understanding the effective depth enable us to build better models or better utilize current ones?

  2. This paper would benefit from a thorough editing pass for writing style and clarity, especially in the setup section “3.1 Models and Data”:

    Introduction: Please define “effective depth” before using it on L060.

    Section 3.1:

    A. The phrase “endpoint” is used multiple times in Section 3.1 before it is defined. Can you make it clear earlier what the “intervention endpoint” is and how it is determined?

    B. This sentence on L151-154 is unclear and especially difficult to parse:

    Endpoint comparisons additionally require a donor-directed effect in at least one layer of each task. This gate separates intervention effects from pre-existing task errors, and it is what makes the per-model item counts differ across analyses.

    Up until this point, It has not been defined what “endpoint comparison” you are looking for, or what a “donor-directed effect” is. Additionally, “in at least one layer of each task” should be reworded to make it clear that for each task, you intervene in every layer, and you need at least one layer’s intervention to be successful for both tasks.

    I also don’t understand, at this point in the paper, what it means that the “per-model item counts differ across analyses.”

    Can you rewrite this?

    C.

    The question-type analysis of Section 4.3 uses a separate set of 918 items from the same split, which are not paired and carry no donor image.

    Probably you mean “which are not paired with any donor image”?

    D.

    Questions are generated from the annotated cells of the table, so each item carries its evidence cells with cell-level bounding boxes and a gold answer computed from them.

    I think “carries” is an LLM-ism, I think you mean that “each example is labeled with its evidence cells, cell-level bounding boxes, and a gold answer”.

    E.

    Operation labels describe what a question asks.

    It may be possible to omit this sentence, I’m not sure what point this is trying to make.

    Misc. A. L352: “Without a threshold, all eight models separate the roles.”

    Can you change this to a paragraph heading that more clearly states what is actually being compared, which is the area under DpD_p?

    B. L408-414: I have no idea what the paper is trying to say about the “different things” that removal and replacement are trying to measure. Can you rewrite to clarify?

Reviewer_okFgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Paper proposes a framework to understand visual evidence of a cell in a table for tabular document question-answering by considering two aspects, (a) Lookup: read out the answer and, (b) Compute: used as an operand with values from other cells, and when removing the cell stops mattering in the model.

Critical strengths
  1. The paper works on an important problem in interpretability of VLMs for tabular document input, where the understanding the difference between how model process OCR inputs differently for (a) seeing, i.e. just extracting the direct OCR in tables, vs (b) using, i.e. extraction the text information and further using it as computation, is studied
Critical weaknesses
  1. The paper is very poorly written and is very difficult to follow. Framework section, especially section 3.2 about mapping table cells to visual sites: it is very unclear on how visual input is processed and edited for different models, as each model has its own resolution. Other details such as visual token count per page is also not mentioned.
  2. Very vague details about dataset construction and usage.
  3. Overall, the paper needs a lot of revision in terms of design choices of the framework, missing details, clarity of the paper, and readability of the paper.
Questions for authors
  1. Most of the explanations in 4.1 section are just read-outs of the figures and tables. It doesn’t provide any additional information or insights.
  2. Figure 3, caption is cut-off by figure 4.
  3. Can you please explain what is the significance after understanding that few operands do not matter after certain layers? How does this help training or evaluation?
  4. There is no gradual flow of events. It is hard to understand what happens to the cell value from original image tokens in the page to the final answer, in both settings. Maybe a diagram explaining each stage might provide better understanding.
Reviewer_Zn11OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a question in visual table understanding on multimodal LLMs: whether the model process every cell to the same depth regarding the functional role of the cell. Specifically, the paper argues that a cell can serve as two roles, LOOKUP(the value is just the answer) or COMPUTE(the value serves as intermediate computation information). This paper focuses on identifying the difference in processing patterns regarding these two roles by designing activation patching and found that when serving as operand, the cell value stops influencing the model's output at an earlier layer compared to when it serves as the answer. This finding is observed with log-probability criterion for 8 models and full answer generation for 6 models.

Critical strengths
  1. The experimental design in this paper is reasonable and relatively comprehensive to justify the conclusion (though I have some concerns regarding the problem setup, see weakness). Specifically, authors construct paired questions based on the same table and target cell where the target cell serves as two different functional roles (LOOKUP and COMPUTE) with minimal variations on the visual input. The experiment also covers a wide range of MLLMs and additionally incporate few complementary protocol such as layer-wise activation patching and linear probing to strengthen the results.
Critical weaknesses
  1. My main concern about this paper is the problem scope. The research question studied in this paper is: whether the functional role of a table cell as an answer (termed as LOOKUP) or an operand (termed as COMPUTE) affects how deeply its visual representation is causally involved in MLLM predictions. This research question imposes a few restrictions: 1) the input is a visual table; 2) the main study subject is a value cell in this visual table and 3) the functional roles of a cell are limited to two specific cases, namely serving directly as an answer (LOOKUP) or as an operand in simple arithmetic computation (COMPUTE). Each of these restrictions by itself may be reasonable but putting together of them makes the research problem way too specific. It is also unclear to me why this particular setup and the distinction between these two roles are important.

  2. Even with this problem setup, the research insight is still incremental to me given the existing literature [1, 2, 3]. While this paper properly cited these existing works, the main conclusion basically follows what has been already established in related domains. It is unclear to me what additional understanding this paper provides beyond the existing conclusions except justifying in this specific problem domain (visual table understanding). The study is also limited to direct answer generation, i.e, only one or a few answer tokens are decoded right after the question prompt. However, if models are allowed to generate intermediate steps of reasoning (which is quite natural in reasoning questions regarding arithmetic computation), whether the conclusion regarding the distinction between LOOKUP and COMPUTE roles still holds is unclear. And this direct answer generation setup gives an additional restriction to the problem of study in this paper.

[1] Stolfo et al., A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis.

[2] Mamidanna et al., All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokens.

[3] Wang et al., When Token Pruning is Worse than Random: Understanding Visual Token Information in VLLMs.

Questions for authors
  1. Could authors clarify on the significance of the problem they studied in the paper? For example, why the distinction between the model's processing pattern of LOOKUP and COMPUTE an important problem to study and what broader understanding of multimodal reasoning does this work provide given the existing literature?

  2. Have the authors considered allowing models to generate intermediate reasoning steps for this study and would it alter the main conclusion of this paper?

  3. The study has some correctness filtering process that only uses the examples that the model can get correct answers for analysis, which makes sense to me. However, one concern is that the current dataset seems to be challenging for small models (e.g, 8B scale ) and only a very few number of examples are kept. This results in very small evaluation sets for some models and the evaluation sets are also different across different models . Would a simpler synthetic setting better control the input complexity and provide more consistent evaluation setups for this study?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.