- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper is a mechanistic examination of how VLMs perform table visual question answering. Specifically, it traces how the relevant cells’ information are processed within the model by quantifying the information’s ``intervention endpoint”, which is defined as the last layer at which patching in an alternative hidden state will change the model’s output. The paper shows that the effective depth varies based on the role of the cell value: when a value is read out in the answer (a LOOKUP operation), it has a deeper effective depth than when the value is used as an operand (a COMPUTE operation).
Critical strengths
- The experimental design is sound: the taxonomy of “seen” vs. “used” cells is a useful one, and the method of validating the intervention site and thresholds are thoughtful.
- The experimental results are thorough: The paper’s conclusions are drawn from experiments on 8 different models. Additionally, confidence intervals are provided for substantive claims as in Table 1.
Critical weaknesses
- I’m not sure what the practical value of this paper has. It presents an observation about the relative intervention “depth” of information, but is unclear what the utility of this observation is. How can this observation help us improve future models, or how we use current ones?
- The paper’s presentation, specifically the writing, needs substantial improvement. Specifically, the setup sections would benefit from editing passes to improve style and clarity.
Questions for authors
-
What are the practical implications of the findings in the paper? How does understanding the effective depth enable us to build better models or better utilize current ones?
-
This paper would benefit from a thorough editing pass for writing style and clarity, especially in the setup section “3.1 Models and Data”:
Introduction: Please define “effective depth” before using it on L060.
Section 3.1:
A. The phrase “endpoint” is used multiple times in Section 3.1 before it is defined. Can you make it clear earlier what the “intervention endpoint” is and how it is determined?
B. This sentence on L151-154 is unclear and especially difficult to parse:
Endpoint comparisons additionally require a donor-directed effect in at least one layer of each task. This gate separates intervention effects from pre-existing task errors, and it is what makes the per-model item counts differ across analyses.
Up until this point, It has not been defined what “endpoint comparison” you are looking for, or what a “donor-directed effect” is. Additionally, “in at least one layer of each task” should be reworded to make it clear that for each task, you intervene in every layer, and you need at least one layer’s intervention to be successful for both tasks.
I also don’t understand, at this point in the paper, what it means that the “per-model item counts differ across analyses.”
Can you rewrite this?
C.
The question-type analysis of Section 4.3 uses a separate set of 918 items from the same split, which are not paired and carry no donor image.
Probably you mean “which are not paired with any donor image”?
D.
Questions are generated from the annotated cells of the table, so each item carries its evidence cells with cell-level bounding boxes and a gold answer computed from them.
I think “carries” is an LLM-ism, I think you mean that “each example is labeled with its evidence cells, cell-level bounding boxes, and a gold answer”.
E.
Operation labels describe what a question asks.
It may be possible to omit this sentence, I’m not sure what point this is trying to make.
Misc. A. L352: “Without a threshold, all eight models separate the roles.”
Can you change this to a paragraph heading that more clearly states what is actually being compared, which is the area under ?
B. L408-414: I have no idea what the paper is trying to say about the “different things” that removal and replacement are trying to measure. Can you rewrite to clarify?