acceptodds
Under review as a conference paper at ICLR 2027

The Gordian Knot for VLMs: Diagrammatic Knot Reasoning as a Hard Benchmark

Abstract

A vision-language model can fail a diagram question by misreading the diagram or by failing to operate on what it read. KnotBench pairs a corpus of 858,318 rendered diagrams of 1,951 prime-knot prototypes with 2,000 evaluation items in 14 tasks, labeled from the knot census and stored planar diagram (PD) codes. Its equivalence and move questions come as images and as PD codes, to separate these two sides of a perception-operation gap. We evaluate Claude Opus 4.7 and GPT-5 at two thinking settings each, under unequal output caps. Of the 56 (task, configuration) pairs, 22 (18 outside C1, whose baseline is 0) are at or below the uniform random baseline, and on 6 of 14 tasks no configuration scores above 65%. All configurations count crossings worse than a constant answer and match images to PD codes at chance. Pooled over the paired questions, they score 14.3 to 33.8 points higher on PD codes than on images. Thinking adds 2.0 points for Claude and 5.4 for GPT-5, mostly on three PD-code tasks, two of which text rules solve. The data establish a failure to read structure from our drawings; since real-step move labels also follow from crossing counts, whether the models simulate moves remains open.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, benchmark, diagrammatic reasoning, knot theory, topological reasoning, spatial reasoning, perception-operation gap, Reidemeister moves, reasoning models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ksuSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces KnotBench, a benchmark for evaluating vision-language models on diagrammatic knot reasoning. The benchmark contains 2,000 evaluation questions across 14 tasks, including knot equivalence, move prediction, crossing identification, and image-symbol matching. Experiments on Claude Opus 4.7 and GPT-5 show that models generally perform better on symbolic inputs, while additional thinking provides limited improvements.

Critical strengths
  1. Knot diagrams provide a challenging setting for evaluating structural understanding and topological reasoning, with mathematically defined relationships and labels.

  2. The authors investigate shortcut-based solutions, compare performance against simple baselines, and acknowledge several limitations in their evaluation.

Critical weaknesses

Previous work on scientific diagram, like Math Blind (Sun et al., ICLR 2026), Diagram-MMU (Bo et al., ECCV 2026), understanding has already shown that visual perception and structural understanding are major limitations of current VLMs. The main conclusions of this paper appear largely consistent with these findings. It remains unclear what new insights knot diagrams provide beyond demonstrating the same limitation in another domain.

The paper lacks a systematic human expert evaluation of the visual correctness and answerability of the generated diagrams, and it is insufficient separation of perception and reasoning. for example, some symbolic tasks can be solved using simple shortcuts, like counting crossings or checking tuple order, without performing genuine topological reasoning. In addition, the image and symbolic variants use separately sampled examples. These factors make it difficult to attribute the performance gap solely to visual perception.

The experiments evaluate only two proprietary model families, with different output-token budgets. The paper mainly demonstrates that existing models struggle with knot diagrams, and no solutions are experimentally evaluated.

Questions for authors

Refer to critical weaknesses

1 comment
AuthorsOpenReview ↗

Reply to Reviewer ksuS

Thank you for the report. Your three points (novelty, the check of the drawings, and the range of models and remedies) shaped most of the revision. Section, figure and table numbers refer to the revised PDF.

W1. What do knot diagrams add beyond Math Blind and Diagram-MMU? We agree with the conclusion of that work, that perception limits current models on diagrams, and we now discuss both papers and IsoBench (Sec. 7, with a comparison table in App. P). Knot diagrams let us locate a failure step by step on the same items. Because every drawing is stored with the PD code of the diagram drawn, the same question can be posed on the drawing and on its code, and what a model read can be scored at each step. This separates a failure to read from a failure to operate, for instance to simulate a Reidemeister move, which a perception score alone cannot do (Table 2):

  • Given PD codes, Claude Opus 4.7 and GPT-5 do not reliably tell whether two diagrams are one R3 move apart, while Claude Opus 5.5 is right on 95%. On drawings of the same pairs Opus 5.5 scores 26%, below the 50% of a constant answer, and its transcriptions place that loss in reading.
  • Opus 4.7 and GPT-5 read a crossing shown alone at 84–99%, so they can see how a crossing is drawn, but read the same crossings inside the whole drawing at 47–54% (chance 50%). Opus 5.5 reads 98.7% of crossings inside the drawing but writes the exact diagram for 10 of 60 drawings.
  • With thinking, both submitted models tell whether two PD codes show the same diagram (95% and 85%, against 54% and 53% without). GPT-5+T also thinks on drawings of the same pairs and scores 54%, so thinking helps the operation, not the reading.

W2. Human evaluation of correctness and answerability; shortcuts; separate samples; attribution.

Correctness. Every new drawing is checked by computer against its PD code: the over and under strands at each crossing are read back from the pixels and compared with the code. For the single-crossing set, 677 of 678 candidate crossings agree with the key, one is unclear and was left out, and none disagrees.

Answerability. We report no human baseline. The models themselves show that the crossings can be read from the pixels: the four submitted configurations name the upper strand at 97.4–99.5% of 191 crossings cut out and enlarged and at 84.3–89.5% cut out at the original size, and Opus 5.5 reads 98.7% of all numbered crossings inside the drawings and every crossing of 88 of the 100 drawings.

Shortcuts. Two new tests remove the shortcuts you name. In the rebuilt A3 the two diagrams use different layouts and the negatives differ in two crossings of opposite sign; in the balanced move test both diagrams have the same crossing count and writhe. Rules on counts, writhe, signs, tuple order and the closeness of the two codes score 44 to 56% on both (Sec. 4.2).

Separate samples. Every comparison of drawings with codes in Sec. 5.4 uses the same diagrams on both inputs. On the same-diagram twins of the submitted tasks the pooled PD-code advantage is 13.8 to 41.2 points, with every interval above zero.

Attribution. We agree that not every gap is due to perception alone, and the paper now separates two hypotheses. Where a model performs an operation on codes, drawings of the same items cost 31.0 points (GPT-5+T, rebuilt A3) and 69.0 points (Opus 5.5, move test). Where the operation itself fails, as on the move test for Opus 4.7 and GPT-5, we say so and do not attribute the failure to perception (Sec. 5.3).

W3. Two proprietary families, unequal budgets, no solutions evaluated.

Models and budgets. Seven models and configurations from four vendors now answer the diagnostic split (Claude Opus 5.5 a subset of it), two of them open-weight. All new runs have a 64,000-token cap (32,768 for the open-weight models, the most their providers offer), no re-queries and a logged model identifier.

Solutions. We evaluated six remedies on the submitted models (Sec. 5.2). Cutting a crossing out of the drawing restores its reading, from 47.1–53.9% inside the drawing to 84.3–89.5% at its original size. None of the others restores the reading. Twice the resolution raises exact counts by more than two points only for GPT-5, and with every crossing circled exact counts are 13–31%. A worked example moves the over-strand probe by −2.2 to 2.9 points, and maximum effort leaves A2-I at 65% against 68%. Transcribing first is limited by transcriptions that never have the right shadow.

If the revision addresses your concerns, we hope you will consider updating your rating.

Reviewer_XJt5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces KnotBench, a benchmark meant to separate two sources of VLM failure on diagrams: reading structure from pixels, and operating on that structure once it is read. The corpus has 858,318 renders of knot diagrams. They come from Metropolis-filtered random Reidemeister walks starting at 1,951 census prime knots and their mirror images, and each render stores its PD code. From 261 test prototypes the authors build 2,000 items in 14 tasks. Equivalence (A0 to A3) and move prediction (B0) are posed both on images and on PD codes. C0 and C1 ask for the crossing count and the DT code of a drawing, and D0 and D1 ask whether a PD code matches a drawing. Claude Opus 4.7 and GPT-5 are evaluated with and without thinking, one sample per item, and compared against random, constant, text-rule and oracle baselines. Pooled over the paired tasks, PD-code accuracy exceeds image accuracy by 14.3 to 33.8 points. Thinking gains come mostly from PD-code tasks that text rules solve. A post-hoc audit found that the renderer had switched every crossing in the A to C images. The authors then corrected 25 labels, relabeled 84 A0 items, rebuilt D0 and D1, and documented cue rules that solve several tasks.

Critical strengths
  1. The evaluation on the benchmark provides evidence that the tested models cannot read crossing structure from the drawings. On C0 every configuration counts crossings less accurately than a constant answer. On A2 the models score 45 to 67% on images but 96 to 100% on PD codes for the same question. All labels can be checked exactly with Regina and SnapPy, and the authors did check them.

  2. The authors also report about various problems they found. They report:

    • the renderer bug
    • the wrong amphichiral list
    • B0 items labeled NOT-CONNECTED that are actually one move apart
    • the effect of their re-query policy
    • the missing model snapshot IDs for the May run
    • the cost

They also compare each task score with constant, text-only and oracle rules.

Critical weaknesses
  1. The PD-code tasks does not test the paper's main hypothesis, that the models fail to reason about a diagram's structure even when it is given as a PD code. Several of these tasks can be answered by simple rules that follow from how the items were built:

    • A3-S: only the positive items contain a crossing list that is out of ascending order, so checking the list order gives 100%.
    • B0-S: the move labels were assigned from the change in crossing count, and the prompt describes each move by that change. Comparing the number of tuples in the two codes gives 96%. Of the 22 items where the count does not change, 20 are R3, so this subset cannot separate move simulation from a default answer.

    As a result, the PD-code results cannot show whether the models reason about structure, and the main hypothesis remains untested, as the paper's discussion notes.

  2. The evaluation is limited. Only two closed models are tested, with one sample per item, different output limits across configurations, and no human baseline. The authors list in Section 6 the experiments that would separate the possible causes (larger renders, images with crossings marked, a worked example, a human baseline) but run none of them. So the paper cannot say whether the models miss crossings because of resolution (strokes are 2.8 to 5.1 px), tracing, or misreading the gaps. Claude+T writes 50 tokens or fewer on 994 of its 1,000 image-only responses, so the evaluation says little about thinking on images.

Questions for authors

Q1: Does the PD-code advantage hold if you use the same diagrams in both versions and only count tasks with no known text shortcut? Examples are A0 and the B0 items where the crossing count does not change. Q2: Two experiments would help identify some failure. First, ask the model to write the PD code of the image, then answer the PD-code question using its own code. Second, give the PD code as an image of text. Running even one of the controls you list in Section 6, such as images with the crossings marked, would also help. Q3: Adding an open-weight VLM, another frontier model, and several samples per item would make the results more convincing.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer XJt5 (Part 1 of 2)

Thank you for a precise report. The two experiments you proposed in Q2 and the controls you named, except a human baseline, are now in the paper, and we answer your question about resolution, tracing and gaps. Section, figure and table numbers refer to the revised PDF.

W1. The PD-code tasks do not test the hypothesis. Agreed: the submitted -S tasks could not. We built two tests that none of the simple rules we tried solves, and checked every label independently (Sec. 4.2). The rebuilt A3 shows two diagrams in different layouts, and its negatives are the same diagram with two crossings of opposite sign switched. The balanced move test, the test of move simulation that our Section 6 proposed, asks whether two diagrams with the same crossing count and writhe are one R3 move apart; every label was checked by enumerating all R3 moves with Regina. Rules on counts, writhe, signs, tuple order and the closeness of the two codes score 44 to 56% on both. The result depends on the operation (Sec. 5.3). On the rebuilt A3, which asks whether two codes show the same diagram up to relabeling, both models are indistinguishable from 50% without thinking (52.5 and 53.5%) and succeed with thinking (95.0 and 85.0%). On the balanced move test, no configuration of Opus 4.7 or GPT-5 is significantly above 50%, the score of a constant answer (48.0, 54.0, 44.5 and 57.0%; smallest Holm-corrected p = 0.387; the upper ends of the 95% intervals lie between 51.4 and 66.3%). Claude's score comes from answering R3 on 180 of 200 pairs. Claude Opus 5.5 is right on 95% of the 100 pairs it answered. The paper now reports the operation hypothesis per test. For the submitted models it survives the move test and, once they think, fails for the comparison.

Q1. Same diagrams, no known shortcut. (Answered here because it follows from W1.) Yes, wherever the operation is within the model's reach, and we lead with the contrasts in which the model thinks on both inputs (Sec. 5.4). GPT-5+T scores 85% on the rebuilt A3 from codes and 54% from drawings of the same 100 pairs (31.0 points [19.4, 44.0]; about 15,289 and 4,565 output tokens per item). Opus 5.5 scores 95% on the balanced move test from codes and 26% from drawings of the same pairs (69.0 points [59.6, 78.0]; 6,339 and 3,054 tokens). Claude+T's gap of 49.0 points on the rebuilt A3 comes with 17,979 tokens per code item and 11 per drawing, so that contrast also compares thinking with not thinking. For Opus 5.5 we located the loss. On its own PD codes of the 60 drawings of 30 pairs, an exact R3 search is right on 14 of 30 pairs, and on 1 of the 15 that are one move apart; on the true codes the search is always right. The reading loss, the difference between the two, is 53.3 points [36.1, 69.8]. The operation loss, the search minus the model's own answers on its codes, is −6.7 points [−16.7, 0.0].

Where the operation itself fails, drawings can still be worse. On the move test Claude, Claude+T and GPT-5+T score 35, 30 and 40% from drawings, below the 50% of a constant answer, because they name a move that changes the crossing count on 38, 45 and 16 of 100 pairs whose diagrams have equal counts. On the A0 twins no model exceeds the best non-visual rule from codes (72.0%); from drawings GPT-5, GPT-5+T, Qwen3-VL and GLM-4.6V score 31.0, 21.0, 23.0 and 21.0 points lower, and Claude and Claude+T show no significant difference.

AuthorsOpenReview ↗

Reply to Reviewer XJt5 (Part 2 of 2)

W2. The evaluation is limited. Point by point:

  • Two closed models. Three models were added: Claude Opus 5.5 on a subset of the new tests (Table 2), and the open-weight Qwen3-VL-235B and GLM-4.6V on all of them, the latter two from two further vendors through a hosted API with pinned providers. The two open-weight models also answered the 2,000 submitted items (46.1 and 46.5% over the 13 tasks).
  • One sample. Claude and GPT-5 answered the core sets (the twins, the re-queried images, the rebuilt A3 and the balanced move test) three times; per-task accuracy moved by at most 3.0 and 6.0 points. Claude+T and GPT-5+T answered the A2 twins and images twice. Everything else is one sample, which the limitations state.
  • Different caps. All new runs had a cap of 64,000 tokens (32,768 for the open-weight models, the most their providers offer), no re-queries and a logged model identifier. Of the thinking responses to PD-code items, 27 reached the cap and count as wrong.
  • Human baseline. We report no human baseline. Legibility rests on two things: every new drawing is checked against its code from the pixels, and the submitted models read the same crossings when these are shown alone (see below).
  • Controls of Section 6. All reported except the human baseline: twice the resolution, crops, circled crossings and a worked example. We also forced maximum effort.
  • Resolution, tracing, or gaps? We showed the same 191 crossings five ways (Table 2, Sec. 5.2). Alone and enlarged, the four configurations read 97.4–99.5%, and alone at the original size 84.3–89.5%, so the gaps are legible, the convention is known, and resolution alone is not the limit. Enlarged four times with the neighboring strands in view, they read 85.9–95.3%. Inside the whole drawing at its own scale they read 49.7, 49.2, 48.7 and 63.9% with one crossing framed, 49.2–52.4% with one number asked, and 47.1–53.9% of these crossings with all numbers asked. The whole drawing at its own scale defeats the reading, and asking for many crossings at once is not the cause. Tracing fails separately. None of 1,040 transcriptions has the right planar shadow (the diagram without over and under), and Opus 5.5, which reads 98.7% of crossings in place, still gets the shadow wrong on 28 of 60 drawings.
  • Claude+T does not think on images. True under the new cap as well, so we forced it. At maximum effort with a describe-first instruction it writes about 967 tokens per item and scores 65% on A2-I against 68%. GPT-5+T does think on images and still scores 49.0% on A2-I, the same as GPT-5 (49.0%).

Q2. The three experiments. (i) Transcribe, then answer from the code. Run on the A2 twins and the move test. For the four configurations no transcription has the right shadow, and an exact operator on their own codes scores 6–56% on A2 and 0–30% on the move test, against 100% on the true codes. On A2 the models' own answers from these codes are no better (6–54%). Because A2's operation is only a count comparison, this locates the failure in reading and does not test the operation; the rebuilt A3 and the balanced move test do, and for Opus 5.5 the full decomposition is in Q1. (ii) PD code as an image of text. The four configurations copy both codes exactly on 85–99% of items and answer at 97–100%, against 49–68% for drawings of the same diagrams, so image input as such is not the obstacle. (iii) Marked crossings. With every crossing circled, exact counts are 13–31%, but the circles alone on a blank canvas are counted correctly only 30–54% of the time, so counts partly measure the counting of many marks. We therefore rest the reading claims on tests other than counts.

Q3. Open-weight model, another model, several samples. See W2 above. Both open-weight models are also more accurate on codes than on drawings of the same diagrams (by 31.4 and 19.2 points, pooled) and stop earlier on the steps of Table 2: Qwen3-VL reads a crossing alone (88.0%) but not with neighbors (54.5%), and GLM-4.6V scores 65.4% even on one enlarged crossing.

If the revision addresses your concerns, we hope you will consider updating your rating.

Reviewer_kxX2OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new VLM benchmark of 2,000 questions about knots, which are developed from 858,318 rendered images of 1,951 prime knots. One of the main capabilities it seeks to test is whether or not VLM's struggle on visual math problems because of issues understanding the image (perception) or reasoning about the problem (operation). The authors' stated main result from the evaluation on Opus 4.7 and GPT-5 is that the pain point for VLM's is perception based on improved performance on PD codes as well as evaluation of visual grounding questions.

Critical strengths
  1. Using knot diagrams that have an accompanying PD code as a way to tease apart what the authors call perception and operation in VLM's is interesting and potentially useful.
  2. Clearly a lot of effort has been put into creating and curating corpus and evaluation items, trying to establish good baselines for performance, and evaluating model performances on different metrics.
Critical weaknesses
  1. I think the exposition of what is actually in the benchmark can be improved a great deal.

    a) I would strongly recommend taking Figure 9 out of appendices, adding to it to also visually show how this corpus item is then developed into a benchmark/evaluation item and possibly emphasizing this perception-operation gap, and making it Figure 1. Most people looking at this paper will skim it and without a better figure illustrating what is going on, they are not going to come away with a strong idea of what the benchmark is.

    b) I think that the abstract has a bunch of numbers of corpus vs. prime knots vs. eval items vs. tasks that are confusing for someone that just reads the abstract--the most important numbers in this exposition are number of corpus items and number of questions in the benchmark.

    c) Also, I think starting off the abstract with the corpus number is a bit misleading for understanding size of benchmark, since the question generation code is not mentioned as being released.

    d) In 4.1 I would like some intuition for why these families are the most relevant, how they are all distinct, and how they relate to broader VLM capabilities.

  2. The benchmark utility, whether for the perception-operation gap or other probing of VLM capabilities, needs to be better articulated.

    a) We already have previous work that demonstrate VLMs do better on text reasoning rather than image reasoning, as cited in related work. What is the benefit of showing this on knot diagrams with this particular setup? How are they uniquely useful such that researchers will use your benchmark in the future? I think explaining these and stating the difference with different knot benchmarks more clearly would be helpful for the manuscript.

    b) To clarify the point, is the point of this benchmark to make VLM's better at math in general, or at knot theory, or to help study the perception-operation gap, or all of the above? Right now the emphasis feels more on this being a hard task which is interesting but not sufficient for being an actionable benchmark for other researchers.

    c) Were there reasoning failure modes particular to knots that were observed in the evaluation (typical incorrect answers or in the chains of thought) that were particularly insightful, that can explain the unique benefit of knot diagrams for general VLM study? Some of these are in the appendices but perhaps some brief mention in the manuscript would be helpful.

    d) If the point is to study this perception-operation gap, why are the knot diagrams separately sampled between the -I and the -S variants? I think this weakens the attempt to isolate these separate steps in a VLM's problem-solving process--we don't know if there's something that differs in the difficulty distribution of the knot diagrams. And to compound this weakness, I don't see an explanation for this experimental choice.

    e) What do C and D add to the benchmark evaluation for perception-operation? The interpretation of the A and B tasks are clear in Figure 2 relative to these.

    f) C1 being included as a benchmark task doesn't make sense while there exists this issue of the drawing having more crossings than the census code on 94/100 items. The model has no way to get a correct answer while correctly describing the diagram, and so C1's inclusion affects the total score negatively.

    g) The authors acknowledge "cue rules" as ways for the evaluated model to achieve a task more easily than intended, such as on A3-I, where the layouts are the same even though they could be perturbed to more clearly test for this "same planar diagram up to relabeling" capability. Perhaps in this case, a few items that do further perturbation to the shape would be helpful to support the case that this benchmark tests what it says it does.

These are a lot of points but I am looking forward to hearing the author's thoughts.

Questions for authors
  1. Why are two different reasoning levels being compared here? I think that the impact of reasoning effort is a separate study question that is somewhat orthogonal to the main goal of studying the perception-operation gap and it is taking up space in the main paper.
  2. How do the takeaways about budget and output caps fit into the main perception-operation narrative?
  3. The abstract ends with a thought about whether models simulate moves. How would a study of this be implemented on the current benchmark beyond what is described in 6 (is this benchmark a viable platform for really delving into this topic)? Also, since you don't do substantial work here or link it to a broader goal for VLM's I wouldn't include it in the abstract.
  4. Will the whole corpus and evaluation-generation code be released?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer kxX2 (Part 1 of 2)

Thank you for a detailed and constructive report. We answer every point below; section, figure and table numbers refer to the revised PDF.

W2b. What is the benchmark for? We start here because the answer frames the rest. KnotBench asks why a vision-language model fails on a diagram: because it cannot read the structure from the drawing, or because it cannot operate on that structure, for instance simulate a Reidemeister move (your Q3). Every drawing comes with the PD code of exactly the diagram drawn, so the same operation can be posed on the drawing and on its code, and each step of reading can be scored on its own. Table 2 gives each model's profile over these steps, and the models stop at different ones. GLM-4.6V names the upper strand of one enlarged crossing at 65.4%, where chance is 50%. Qwen3-VL reads a crossing alone (88.0%) but not with neighboring strands in view (54.5%). Claude Opus 4.7 and GPT-5 read a crossing alone or with its neighbors (84–99%) but the same crossings inside the whole drawing at 47–54%. Claude Opus 5.5 reads 98.7% of crossings inside the drawing but writes the exact diagram for 10 of 60 drawings. The profile also shows which remedy is worth trying: for the four submitted configurations, cutting a crossing out of the drawing restores its reading, even at the original size. Twice the resolution, circled crossings, a worked example and more thinking do not restore it. The diagnostic split and a script that prints this profile for a new model are in the updated supplementary material (Sec. 6).

W1a. Figure 1. Done in part. Panel (a) of the new Fig. 1 takes over the content of the corpus-construction figure you point to: a census diagram, random Reidemeister moves, and the resulting diagram as a drawing and as its PD code. Panel (b) turns a drawing into a test: one crossing asked in the drawing and as an enlarged cutout, with the accuracy on each. Panel (c) places every model on seven tests from reading one crossing to operating on drawings. The figure shows no item of the 14 submitted tasks; their formats are in App. Q, and the full corpus-construction figure stays in App. I.

W1b, W1c. Abstract. Done. The abstract opens with the question, gives only the size of the evaluation set, and reports three results instead of counts of pairs at baseline.

W1d. Why these families. Done (Sec. 4.1, first paragraph). Family C asks only for a reading of one drawing, and Family D asks the model to compare its reading with a given code. The image variants of A and B need a reading and then an operation, and their PD-code variants the operation alone. Each step uses an ability that other diagrams also call for: telling which of two lines passes in front, following a curve through crossings, counting many similar marks, and comparing two structured descriptions. Sec. 4.2 adds a test for each step (Table 1).

W2a. What knots add to earlier text-versus-image comparisons. Every intermediate reading can be checked exactly against the code of the drawn diagram: the upper strand at each crossing, the crossing count, and a full transcription. That lets us score reading, operating on a given code, and their combination separately on the same diagrams. IsoBench poses matched image and text inputs but does not score what was read from the image. Math Blind and Diagram-MMU test perception of diagrams but do not pose the question on the symbolic source alone. Sec. 7 now says this, and a table in App. P compares the benchmarks.

W2c. Failure modes. Done (Sec. 6, "How the models fail"). The models that cannot read a drawing often fall back on a fixed pattern or a familiar object. The over-strand probe asks which strand, the horizontal (H) or the vertical (V), is on top at each numbered crossing. Claude+T answers it with a strict alternation of H and V on 80 of 100 drawings, none of whose keys alternates. On the balanced move test, whose two diagrams always have the same number of crossings, the models name a move that changes the count on 8 to 49 of 100 pairs of drawings. Opus 5.5 answers 13 drawings of 13 to 19 crossings with a code of at most eight crossings, seven of them the standard code of the trefoil.

AuthorsOpenReview ↗

Reply to Reviewer kxX2 (Part 2 of 2)

W2d. Separately sampled -I and -S items. Agreed, and fixed. Every image item stores the PD code of the diagram it draws, so we posed A0, A1, A2 and B0 on those codes and asked the image items again in the same week under one cap (the "same-diagram twins", Sec. 5.4, Fig. 2). The PD-code advantage on A2 is 32.0 to 51.0 points for the four configurations, and pooled over the four pairs it is 13.8 to 41.2 points with every interval above zero, so separate sampling did not create it. The rebuilt A3 and the balanced move test use identical pairs on both inputs. There was no design reason for the separate samples: one sampler with the same quotas ran once per variant with its own seed, and we should have drawn one set and shown it both ways.

W2e. What C and D add. They test reading without the operations of A and B. C asks for a reading alone, and D for a reading compared with a given code. Claude+T and GPT-5+T make that comparison between two codes (95% and 85% on the rebuilt A3), so their chance-level D0 points to the reading. The counts are less clean. The models count plain circles on a blank canvas correctly only 30–54% of the time, so C0 also measures the counting of many marks. We therefore rest the reading claims on tests other than counts: the single-crossing tests, the over-strand probe, transcription, and D0 (Sec. 5.2).

W2f. C1. Agreed. C1's key is the census code of the knot, which a correct transcription cannot match on 94 of 100 items. C1 keeps its strict score in the heatmap (App. O) but leaves every aggregate, which now covers 13 tasks (57.3, 59.3, 46.6 and 52.3% for Claude, Claude+T, GPT-5 and GPT-5+T). A direct transcription test replaces it. Asked for the PD code of a drawing, the four configurations write no code with the right planar shadow in 1,040 transcriptions.

W2g. A3-I and its cue. We rebuilt A3 as you suggest. The two drawings of every item use different layouts, and each negative is the same diagram with two crossings of opposite sign switched, chosen so that crossing count, writhe, shadow and knot type agree. Every simple rule we tried scores 44 to 56%. On drawings the four configurations score 46.0–54.0%, inside the range of the rules, so the submitted A3-I scores do not survive distinct layouts. On codes the two thinking configurations do make the comparison (95% and 85%).

Q1. Why two reasoning levels? Because effort turned out to act on one side of the gap only. On the 100 code pairs of the rebuilt A3 that all four configurations answered, thinking takes Claude and GPT-5 from 54.0% and 53.0% to 95.0% and 85.0%, and it brings no reliable gain on drawings of the same pairs (48 to 46% and 49 to 54%). Forcing Claude+T to think at maximum effort and describe the drawing first leaves A2-I at 65% against 68%. The thinking-lift analysis of the submitted items is now in App. O.

Q2. Budget and caps. They no longer carry any part of the argument. The unequal caps were a flaw of the submitted runs, and the main text now mentions them only as a limitation (Sec. 4.3, 5.1 and 6; details in App. K). All new runs used a 64,000-token cap (32,768 for the two open-weight models, the most their providers offer) without re-queries; 27 thinking responses to PD-code items reached it and count as wrong.

Q3. Do the models simulate moves? The benchmark now tests this directly, and the answer is in the abstract and in Sec. 5.3 and 6 ("Mental simulation"). We ran the test our Section 6 proposed: 200 pairs of PD codes, half one R3 move apart, with equal crossing count and writhe and codes renumbered at random. Every label was checked by enumerating all R3 moves with Regina, and no simple rule we tried exceeds 54%. No configuration of Opus 4.7 or GPT-5 is significantly above 50%, the score of a constant answer (48.0, 54.0, 44.5 and 57.0%), and Claude's score comes from answering R3 on 180 of 200 pairs. Opus 5.5 is right on 95% of the 100 code pairs that also exist as drawings, but on 26% of those drawings, and its transcriptions place that loss in reading (Sec. 5.4). So the benchmark can study move simulation now on codes, where it already separates the models, and on drawings once a model passes the reading rows of Table 2 (Sec. 6).

Q4. Release. The updated supplementary material contains the 2,000 submitted items with corrected labels, every item of the diagnostic split (requests, drawings, PD codes, keys and build records), all stored responses of the seven models and configurations, the scoring code, and a script that recomputes Table 2 and prints the same profile for a new model. The full corpus (858,318 images) and its generator are not in the supplement; we plan to release them separately.

If the revision addresses your concerns, we hope you will consider updating your rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

Thank you for three careful reviews. The revision reports the experiments you asked for, within the limits stated below, and the paper is rebuilt around what they show; Appendix N documents every new experiment.

The question. When a vision-language model fails a question about a diagram, has it misread the diagram, or is it unable to operate on what it read, for instance simulate a Reidemeister move (Reviewer kxX2, Q3)? Every KnotBench drawing comes with the PD code of exactly the diagram drawn, so the same question can be posed on the drawing and on its code, and every intermediate reading can be checked against the code. The revision adds a diagnostic split built for this question (Sec. 4.2, Table 1) and evaluates seven models and configurations from four vendors: the four submitted configurations, Claude Opus 5.5 (on a subset of the tests), and the open-weight Qwen3-VL-235B and GLM-4.6V.

Results (Table 2 of the PDF, item counts in App. N; % correct unless a count is given; bold = at least 85%; – = not run):

ClaudeClaude+TGPT-5GPT-5+TOpus 5.5Qwen3-VLGLM-4.6V
One crossing: which strand is on top? (%; chance is 50)
alone, enlarged 8 times99.599.097.497.9–88.065.4
alone, original size86.989.584.388.0–75.950.3
with its neighbors, enlarged 4 times92.795.385.992.7–54.559.7
in the whole drawing, framed49.749.248.763.9–49.246.6
in the whole drawing, one number asked49.249.250.852.4–45.547.1
in the whole drawing, every number asked54.049.153.448.898.746.646.0
The whole drawing
crossing count, exact (%)14.011.04.015.062.01.00.0
PD code with the right planar shadow0/2600/2600/2600/26032/600/2600/260
PD code of exactly the drawn diagram0/2600/2600/2600/26010/600/2600/260
D0: does this PD code match the drawing? (%)49.548.550.046.560.547.043.0
Operating on PD codes (%; simple rules score 46 to 54)
rebuilt A3: the same diagram?52.595.053.585.0–51.551.5
balanced move test: one R3 move apart?48.054.044.557.095.047.045.5
The same pairs as drawings (%; simple rules score 44 to 56)
rebuilt A3: the same diagram?48.046.049.054.051.051.043.0
balanced move test: one R3 move apart?35.030.050.040.026.032.055.0
  1. Operating on a given code. On PD codes, no configuration of Opus 4.7 or GPT-5 reliably tells whether two diagrams are one R3 move apart (44.5–57.0%; none is significantly above the 50% of a constant answer), while Opus 5.5 is right on 95% of pairs. With thinking, both submitted models tell a diagram from a copy with two crossings switched (95% and 85%).
  2. The same pairs as drawings. No model is above the level of simple rules: GPT-5+T falls from 85% to 54%, Opus 5.5 from 95% to 26%. Opus 5.5's transcriptions place that loss in reading (Sec. 5.4).
  3. Where reading fails. Opus 4.7 and GPT-5 name the upper strand of a crossing cut out of its drawing at 84–99%, but of the same crossings inside the drawing at 47–54% (chance 50%). With one crossing framed, three of the four stay at 48.7–49.7% and GPT-5+T reaches 63.9%. Opus 5.5 reads 98.7% of the crossings inside the drawing but transcribes 10 of 60 drawings exactly (Sec. 5.2).
  4. What KnotBench adds. For each model it shows whether the failure lies in the operation or at a step of reading (Sec. 6).

Not included. We report no human baseline. Every new drawing is checked against its code by reading the strands back from the pixels. Opus 4.7 and GPT-5 read the same crossings once these are shown alone.

How the new runs were made. One cap of 64,000 output tokens (32,768 for the open-weight models, the most their providers offer); no re-queries; served model logged. Claude and GPT-5 answered the core sets three times, Claude+T and GPT-5+T the A2 twins and images twice, everything else once. We wrote down how each result would be interpreted before we scored it; Appendix N lists these interpretations and when each was written.

Main changes to the paper.

  • A new Fig. 1; the abstract opens with the question.
  • Sec. 4.2 and Table 1: the diagnostic split.
  • Sec. 5: reading, operating on a code, and both inputs (Table 2, Fig. 2).
  • Sec. 6: mental simulation, and what the benchmark is for.
  • Sec. 7 and App. P: IsoBench, Math Blind and Diagram-MMU.
  • C1 leaves every aggregate; the task table, the heatmap and the details of the submitted items move to App. O and Q.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.