- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces KnotBench, a benchmark for evaluating vision-language models on diagrammatic knot reasoning. The benchmark contains 2,000 evaluation questions across 14 tasks, including knot equivalence, move prediction, crossing identification, and image-symbol matching. Experiments on Claude Opus 4.7 and GPT-5 show that models generally perform better on symbolic inputs, while additional thinking provides limited improvements.
Critical strengths
-
Knot diagrams provide a challenging setting for evaluating structural understanding and topological reasoning, with mathematically defined relationships and labels.
-
The authors investigate shortcut-based solutions, compare performance against simple baselines, and acknowledge several limitations in their evaluation.
Critical weaknesses
Previous work on scientific diagram, like Math Blind (Sun et al., ICLR 2026), Diagram-MMU (Bo et al., ECCV 2026), understanding has already shown that visual perception and structural understanding are major limitations of current VLMs. The main conclusions of this paper appear largely consistent with these findings. It remains unclear what new insights knot diagrams provide beyond demonstrating the same limitation in another domain.
The paper lacks a systematic human expert evaluation of the visual correctness and answerability of the generated diagrams, and it is insufficient separation of perception and reasoning. for example, some symbolic tasks can be solved using simple shortcuts, like counting crossings or checking tuple order, without performing genuine topological reasoning. In addition, the image and symbolic variants use separately sampled examples. These factors make it difficult to attribute the performance gap solely to visual perception.
The experiments evaluate only two proprietary model families, with different output-token budgets. The paper mainly demonstrates that existing models struggle with knot diagrams, and no solutions are experimentally evaluated.
Questions for authors
Refer to critical weaknesses
Reply to Reviewer ksuS
Thank you for the report. Your three points (novelty, the check of the drawings, and the range of models and remedies) shaped most of the revision. Section, figure and table numbers refer to the revised PDF.
W1. What do knot diagrams add beyond Math Blind and Diagram-MMU? We agree with the conclusion of that work, that perception limits current models on diagrams, and we now discuss both papers and IsoBench (Sec. 7, with a comparison table in App. P). Knot diagrams let us locate a failure step by step on the same items. Because every drawing is stored with the PD code of the diagram drawn, the same question can be posed on the drawing and on its code, and what a model read can be scored at each step. This separates a failure to read from a failure to operate, for instance to simulate a Reidemeister move, which a perception score alone cannot do (Table 2):
W2. Human evaluation of correctness and answerability; shortcuts; separate samples; attribution.
Correctness. Every new drawing is checked by computer against its PD code: the over and under strands at each crossing are read back from the pixels and compared with the code. For the single-crossing set, 677 of 678 candidate crossings agree with the key, one is unclear and was left out, and none disagrees.
Answerability. We report no human baseline. The models themselves show that the crossings can be read from the pixels: the four submitted configurations name the upper strand at 97.4–99.5% of 191 crossings cut out and enlarged and at 84.3–89.5% cut out at the original size, and Opus 5.5 reads 98.7% of all numbered crossings inside the drawings and every crossing of 88 of the 100 drawings.
Shortcuts. Two new tests remove the shortcuts you name. In the rebuilt A3 the two diagrams use different layouts and the negatives differ in two crossings of opposite sign; in the balanced move test both diagrams have the same crossing count and writhe. Rules on counts, writhe, signs, tuple order and the closeness of the two codes score 44 to 56% on both (Sec. 4.2).
Separate samples. Every comparison of drawings with codes in Sec. 5.4 uses the same diagrams on both inputs. On the same-diagram twins of the submitted tasks the pooled PD-code advantage is 13.8 to 41.2 points, with every interval above zero.
Attribution. We agree that not every gap is due to perception alone, and the paper now separates two hypotheses. Where a model performs an operation on codes, drawings of the same items cost 31.0 points (GPT-5+T, rebuilt A3) and 69.0 points (Opus 5.5, move test). Where the operation itself fails, as on the move test for Opus 4.7 and GPT-5, we say so and do not attribute the failure to perception (Sec. 5.3).
W3. Two proprietary families, unequal budgets, no solutions evaluated.
Models and budgets. Seven models and configurations from four vendors now answer the diagnostic split (Claude Opus 5.5 a subset of it), two of them open-weight. All new runs have a 64,000-token cap (32,768 for the open-weight models, the most their providers offer), no re-queries and a logged model identifier.
Solutions. We evaluated six remedies on the submitted models (Sec. 5.2). Cutting a crossing out of the drawing restores its reading, from 47.1–53.9% inside the drawing to 84.3–89.5% at its original size. None of the others restores the reading. Twice the resolution raises exact counts by more than two points only for GPT-5, and with every crossing circled exact counts are 13–31%. A worked example moves the over-strand probe by −2.2 to 2.9 points, and maximum effort leaves A2-I at 65% against 68%. Transcribing first is limited by transcriptions that never have the right shadow.
If the revision addresses your concerns, we hope you will consider updating your rating.