- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes to assess cross-modal evidential rupture, a new concept that denotes the loss of support between different entities (i.e., text, figures, tables) in a scientific paper (i.e., do specific modalities support each other properly or not, like a text supporting an equation, or a figure supporting a table). To this end they inroduce RUPTURETRACE, which is a framework for detecting cross-modal evidential rupture. While they extensively describe the general framework, it in practice equates to identifying tables, equations, figures and code that belong to specific sections of a scientific paper, and then designing a score to measure if certain evidential relationships (i.e., figures supporting tables, text supporting equations) are persistent across many subsequent sections, and are actually supported. This support between different modalities is simply instantiated as embedding similarity of a Qwen embedding model. Subsequent evaluations then measure whether the predicted score can be used to predict reviewer scores or paper acceptance, and if it detects artificially induced ruptures.
Critical strengths
The paper adresses an interesting topic and does a very extensive evaluation of different facets of the proposed approach, including 1) ablations testing different encoder backbones, 2) evaluations on both revier score/acceptance alignment and on cotrolled rupture detection, 3) evaluations on which topological regiemes are actually encountered in reality, 4) evaluating the effects of different averaging/matching schemes for multiple entity embeddings in a given section, 5) checking if other factors can predict rupture similarly to the actual full method and 6) evaluating the fragility of the approach with respect to changes in behavior of different parts of the pipeline. This analysis tests many different facets of the proposed approach, but ultimately still has severe limitations described in the next section.
Critical weaknesses
- The method description is held very general, which in theory is a reasonable choice if many components can be exchanged within the general framework. However, the implementational choices are crucial for understanding the general framework at all, since otherwise it is impossible to know what certain steps in the derivation actually do. As a result, the derivation of the general framework is not understandable at all without extensive knowledge of the later sections and especially the appendix. This is a severe downside that makes the paper in its current form barely publishable, independent of the results achieved. These are examples for aspects that are not stated in the method derivation, and would have been crucial for understanding it at all:
- "Anchors" are defined generally as the key scientfic elements ("such as a claim, a methodological step, or an experimental setting") which other modalities are linked to. Knowing precisely what an anchor is in practice is required for assessing whether it makes sense to require that it is linked to multiple modalities, or why continuity in the modality structure across subsequent anchors is meaningful. Without this knowledge, most steps that follow do not make sense.
- Since the specific type of anchor is not stated, the ordering of them is unclear as well. It is stated that they have an order, and throughout the method description we find out that this order also matters a lot. But we never get to know how this would be determined (until the appendix), which makes it impossible to judge whether the method makes sense.
- It is entirely unclear how entities from other modalities are aligned to a given anchor. Different instantiations could here make the method either more sensible or completely incorrect, so leaving this undefined is simply inacceptable at that point. Similar things hold for the map E that maps an entity to its representation.
- The text simply says "Let G* = (M, E*) specify the expected evidential relations" and it is unclear what the "expected relations" are. Later, this proves to be a crucial decision that makes or breaks the whole method.
- Key quantities of the structural analysis are used without definition (e.g., the map H_1 and cycle classes, beta_1, the "one-dimensional incidence complex", etc.)
- At line 183, the paper suddenly talks about "runs" as if it was already established that we now measure persistence of certain properties across "runs" of consecutive anchors. But since this is never established, the following sentences are just confusing to read.
- These previous points are only a small set of the issues. The whole derivation is basically not understandable at all without knowing the actual decisions and instantiations that were later made in the experiments, and that are largely just stated in the appendix.
- The following choices in the actual realization of the method raise substantiall concerns about whether it is sensible:
- The general method relies on analyzing whether different modalities present in a scientific paper support each other, which is modeled by a graph. However, only certain edges in this graph are allowed to exist and these are pre-specified by the authors. This means: It is possible that a text supports a equation, but not that an equation supports a text. Similarly, a figure can support a table, but the framework does not allow a table to support a figure. These choices seem completely arbitrary, which is concerning for two main reasons: 1) The method is completely blind to support relationships not defined by the authors (e.g., an equation supporting a text) and more importantly 2) these arbitrary choices are the key reason that these specific loops that are analyzed by the final metric can even exist at all. With a "proper" graph that allows all modalities to support each other, the whole method would be changes, as (in its current form) its purpose is detecting larger cycles and not small cycles between just two nodes.
- The support between two modalities is simply defined as the similarity between normal embeddings of the two modalities created via a standard embedding model. This does not check for precise entailment at all and simply measures broad semantic similarity which undermines the whole methodological backbone of the proposed procedure.
- There are general properties of the proposed framework that question its soundness, even independent of specific instantiations:
- The final evaluation metric basically evaluates if that same channel (channel here means a specific support relationship like text -> equation, figure -> table, or a larger cycle) exists with a sufficient score for multiple consecutive anchors. In the practical implementation, this means that a paper will only receive a high score if consecutive sections all have the same modality (e.g., multiple sections following each other all have a table). If one section has a table, another section has a figure and a third section has an equation, then this paper can simply not be reliably assessed by this metric. Intuitively this does not make sense at all.
- Larger cycles must exist at a single anchor to contribute positively to the final metric. However, different modalities supporting each other will likely happen across different sections of the paper, which is completely indetectable by the proposed metric.
-
The reproducibility statement states that proofs are contained in appendix A, but none of the proofs for the theorems or propositions are actually contained there.
-
the evaluation does not prove the validity of the method. First of all, the predicted scores are evaluated against reviewer scores and paper acceptance. However, the authors admit themselves that this is not correlated to evidential rupture, which is what they actually try to measure. So this evaluation doesn't prove at all that their method can measure what they claim that it can. Further, they include a benchmark to explicitly test rupture detection, but this benchmark seems to be explicitly built to be well addressable by the proposed system, since the synthetic intervention used to create the rupture explicitly targets what the score is measuring. Since successful rupture detection was not established on real data, the authors ultimately did not manage to prove that their proposed method really detects ruptures. Additionally, baselines that were evaluated against for all experiments seem to score comically low considering that the proposed method performs similar to them or outperforms them, even though the entailment between modalities is computed simply via embedding similarity, which can not possibly be accurate (at least not regarding actual entailment).
Questions for authors
Please feel free to treat all of the weaknesses I listed as questions. But I checked them thoroughly and do not think that they are addressable sufficiently.