acceptodds
Under review as a conference paper at ICLR 2027

Do LLMs Transfer Structural Knowledge? Probing Structural Inference as a Cognitive Capability in LLMs

Abstract

Structural knowledge refers to relational patterns that capture an underlying organization beyond the specific entities and contexts in which those relations occur. While recent studies have examined cognitive biases and reasoning behaviors in large language models (LLMs), whether LLMs can infer such structures from natural language and transfer them across contexts remains unclear. We introduce INSTILL, a framework for INvestigating the STructural Inference in LLMs through tasks that require transferring structural knowledge. Each task presents a reference context and four candidate contexts, only one of which preserves the underlying relational structure of the reference. Drawing on five representative structure families motivated by cognitive science, INSTILL constructs diverse structural patterns and contextualizes them as cognitive scenarios grounded in nine domains spanning spatial, social, comparative, and temporal relations. These tasks require models to infer the latent structure from natural language and generalize it across contexts with different entities, relationships, or domains. INSTILL enables controlled analysis of how structural inference varies across structure families, domain congruency, and increasing structure sizes. Our results show that several recent models maintain stable structural inference across domain shifts and increasing structure sizes, whereas smaller variants of the same model families exhibit larger performance drops and greater sensitivity to domain congruency and local relational patterns.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Large Language Models, structural knowledge, structural inference, cognitive capability, evaluation framework

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_E2fmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether large language models can abstract latent relational structures from natural language descriptions and transfer such structural knowledge to contexts with different surface forms. The authors introduce an evaluation framework that converts various abstract structures into natural language scenarios and asks models to identify the candidate whose structure matches that of a reference context. The experiments examine model performance across different structure types, levels of domain congruency, and structure sizes, comparing both model families and model scales. The results suggest that recent strong models are generally robust in transferring structures across domains, whereas smaller models rely more on surface semantics or local relational cues. The paper also includes generation validation and human auditing to improve task reliability and reproducibility.

Critical strengths
  1. The research question is clear and meaningful, systematically examining structural knowledge transfer as a cognitive capability.

  2. The evaluation framework is well designed, allowing controlled analysis across structure types, relational domains, and task sizes.

  3. The experiments are broad in coverage, and the task-generation process, human validation, and released materials support good reproducibility.

Critical weaknesses
  1. The tasks are mainly constructed and validated using a generative model; despite human spot checks, potential linguistic biases or implicit shortcuts may still exist, limiting ecological validity.
  2. The evaluation relies on multiple-choice questions and overall accuracy, making it difficult to distinguish genuine understanding of global structure from the use of local features or heuristic cues.
  3. The conclusions are primarily based on controlled synthetic tasks, leaving generalization to natural scenarios, longer contexts, or more complex relational combinations insufficiently validated.
Questions for authors
  1. Since the task texts are generated by a generative model, did the authors further analyze how linguistic biases or implicit shortcuts might affect the experimental conclusions?
  2. Beyond the multiple-choice evaluation, did the authors consider additional analyses to distinguish genuine understanding of global structure from the use of local heuristic cues?
  3. Beyond controlled synthetic tasks, do the authors plan to validate the generalization of structural transfer on more natural corpora or more complex contexts?
2 comments
AuthorsOpenReview ↗

Response to comments on potential linguistic biases and heuristic shortcuts

We sincerely appreciate your careful evaluation and valuable feedback on our manuscript. We hope that our responses to your comments below clearly address your concerns.

W1. The tasks are mainly constructed and validated using a generative model; despite human spot checks, potential linguistic biases or implicit shortcuts may still exist, limiting ecological validity.

Q1. Since the task texts are generated by a generative model, did the authors further analyze how linguistic biases or implicit shortcuts might affect the experimental conclusions?

Q2. Beyond the multiple-choice evaluation, did the authors consider additional analyses to distinguish genuine understanding of global structure from the use of local heuristic cues?

We would like to clarify that we have examined potential linguistic shortcuts and their effects on structural inference performance. To investigate potential shortcuts arising from linguistic cues, we compare congruent and incongruent conditions in our main experiments (Section 4.3, Table 2), examining how domain similarity affects structural inference performance. We further evaluate entity-count and embedding-similarity heuristics in Appendix C.1. The results show that such shortcuts do not consistently identify the correct candidate across different domains.

In addition, to further examine whether models rely on local structural cues, we conducted an additional experiment controlling for degree sequences. We compared two conditions: one where all candidates shared the reference’s degree sequence (matched) and another where only the correct candidate did (unmatched). In both conditions, the distractors were non-isomorphic to the reference. GPT-5.4 and GPT-5 maintained high accuracy even when degree sequences were matched, whereas their mini variants showed larger performance declines. This pattern is consistent with our main findings, as models that achieve higher performance on structural inference tasks tend to show smaller performance declines under more challenging conditions. Detailed results and discussion are provided in Appendix C.6.

We acknowledge that linguistic patterns in LLM-generated descriptions may introduce biases that influence model predictions. However, the reliability of an LLM-generated benchmark depends not simply on the use of LLMs, but on whether the generation process is carefully controlled and the resulting data are rigorously validated to ensure their suitability for the intended evaluation [1, 2, 3, 4]. In INSTILL, the underlying structures and correct answers are constructed and verified programmatically, while an independent human audit confirms that the generated descriptions preserve the intended relational information with 99.23% accuracy. Although these procedures do not eliminate all potential linguistic biases, we believe that our task design, programmatic verification, and independent human validation support the reliability of our evaluation across different structures and domains.

[1] Jiang et al., Raising the Bar: Investigating the values of large language models via generative evolving testing, ICML 2025.
[2] Yuan et al., LLM-Powered Benchmark Factory: Reliable, generic, and efficient, ACL 2026.
[3] Huang et al., DataGen: Unified synthetic dataset generation via large language models, ICLR 2025.
[4] Li et al., AutoBencher: Towards declarative benchmark construction, ICLR 2025.

AuthorsOpenReview ↗

Response to comments on multiple-choice evaluation and generalization to more realistic settings

W2. The evaluation relies on multiple-choice questions and overall accuracy, making it difficult to distinguish genuine understanding of global structure from the use of local features or heuristic cues.

INSTILL investigates whether LLMs can recognize and transfer relational structures across different contexts. Importantly, models are not given explicit graphs to compare. They must infer relational structures from cognitive scenarios described in natural-language and identify the same structure despite changes in entities, expressions, and domains. This provides a controlled way to examine the abstraction and transfer of structural knowledge. As discussed in Section 3 (Lines 146-151), structural knowledge lacks a canonical representation, making it difficult to assess directly. Therefore, we adopt a multiple-choice format to evaluate this capability without requiring models to explicitly represent the inferred structures.

Cognitive science-inspired evaluations often assess latent cognitive abilities through observable behavior on controlled tasks, without assuming that the underlying mechanisms are identical to those of humans [5, 6]. For example, CogEval [5] evaluates cognitive maps and planning through tasks that vary underlying structures, without aiming to directly compare LLM performance with humans or establish that LLMs exhibit human-like cognition. ReCogLab [6] also draws on cognitive science experiments to investigate relational reasoning in LLMs, while cautioning that similarities in observed behavior do not necessarily imply identical cognitive mechanisms. In line with this perspective, INSTILL aims to characterize the ability of LLMs to generalize relational structures, rather than establish that they acquire or represent structural knowledge in the same way as humans. Our findings reveal how this ability varies across structural families, domain shifts, and structure sizes, with substantial differences in robustness across models.

[5] Momennejad et al., Evaluating Cognitive Maps and Planning in Large Language Models with CogEval, NeurIPS 2023.
[6] Liu et al., ReCogLab: A Framework Testing Relational Reasoning & Cognitive Hypotheses on LLMs, ICLR 2025.

W3. The conclusions are primarily based on controlled synthetic tasks, leaving generalization to natural scenarios, longer contexts, or more complex relational combinations insufficiently validated.

Q3. Beyond controlled synthetic tasks, do the authors plan to validate the generalization of structural transfer on more natural corpora or more complex contexts?

We designed INSTILL as a controlled evaluation to systematically examine how structural inference and transfer vary across structures, domains, and graph sizes. Our results reveal clear performance differences across these factors, even within relatively simple settings.

We agree that extending the evaluation to more natural scenarios, longer contexts, and more complex relational combinations is an important direction. However, such settings can introduce additional confounding factors, making it difficult to identify which factors contribute to differences in model performance. As an initial investigation, we therefore prioritize controlling these factors to better understand how structural properties and domains affect model performance. Extending this evaluation to more natural and complex settings remains an important direction for future work.

Reviewer_WzQBOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper seeks to study how LLMs understand and transfer structural knowledge. They do this by creating a datasets of programmatically created sentences which correspond to a specific underlying graph structure and then create multiple-choice questions questions asking LLMs to find the same underlying structures. They find that flagship models are generally capable of this task.

Critical strengths

The idea of using programmatically created evaluations to test specific tasks that may correspond to underlying human cognitive capabilities is a good direction. Figures were very clear and informative.

Critical weaknesses

The biggest criticism that I have with this paper is that the scope seems very narrow and incremental from work already done in this space. Boiling down what this paper actually does in my understanding, it takes a technique from prior work (ReCogLab) of creating graphs, translating them to sentences by grounding them in some domain using templates written by an LLM and then testing LLMs with different configurations. In fact, from some of the sample sentences in the figures and the descriptions of the domains, it may actually just be using much of the code and data from that work. And, unlike that prior work, this only really seems to have one type of question: given a graph G, which of these four graphs is it the same as. The scope here just feels very incremental to rise to the level of an ICLR paper.

The next biggest weakness I see is that I am not really sure that the question and the way they ask it really sheds much light on the transfer of structural knowledge observed in humans. Like stated above, the task is really just asking, here is a graph, which of these four graphs is the same graph. This does not seem to me to be very similar to how humans incorporate or transfer structural knowledge. The task as written seems very stilted and unnatural (and, it seems actually maybe ambiguous what the prompt of "underlying structure" is even asking), so I am not really sure what these experiments really tell us about LLMs and potential similarity to humans. It is also multiple choice rather than open ended. So is it really showing transfer of knowledge, or is it really just asking the LLM to compare graphs? Relatedly, I struggled to really see what the purpose of this paper was. I was not very convinced by the comparison to human cognition and even if I was, I am not really sure what takeaways I should be taking from this paper. In general this paper's motivation is just not very clear. You say it "is not intended to compare LLMs with humans or to assess whether their reasoning aligns with human cognition" but then I really struggle to understand what the paper is trying to do.

The details of construction are barely mentioned in the main body of the paper and do not convey a very strong sense of what this dataset really is. Section 3 is dedicated almost entirely to describing the cognitive science background of different things but not so much on details of the dataset. Like, how many nodes are in the graph, how many entities are there, how long are these sentences, what do they actually look like, how many questions is the dataset? What is the source of the data?

I am kind of skeptical of LLM-generated datasets that are then used to evaluate LLMs. It seems that the dataset is a main contribution and this was mostly done by LLMs rather than collecting a set more deliberately or with human annotators. There is human validation, but there are fairly well know language and name biases in LLMs and we know that LLMs treat LLM generated language very differently from humans. So to me it seems inappropriate to consider an LLM-generated dataset an ICLR worthy submission.

For experiments, first, from the node ablation in Figure 6 it really does not seem like these graphs are very large at all. And looking at the prompt, I am also wondering how much work was done on this prompt because at least to me, "underlying structure" seems very vague and if you didn't tell me I might not actually know what it was asking either. Is it possible that with some prompt tuning or in-context examples, more of these models would do better?

Again to the point of motivation, it feels like these are just a lot of experiments that do not seem to drive towards a thesis or analysis or purpose.

Questions for authors

Can you please give more details on the construction of the dataset including the ones I listed above?

5 comments
AuthorsOpenReview ↗

Response to comments on distinctions of INSTILL from ReCogLab and the motivation for our evaluation settings

We sincerely appreciate your careful evaluation and valuable feedback on our manuscript. We hope that our responses to your comments below clearly address your concerns.

W1. The biggest criticism that I have with this paper is that the scope seems very narrow and incremental from work already done in this space. (…) it takes a technique from prior work (ReCogLab) of creating graphs, (…)

Although both INSTILL and ReCogLab employ graph-based representations, they differ in their research objectives, task construction, and empirical analyses. We clarify these distinctions as follows:

  1. Research Objective: ReCogLab primarily evaluates reasoning within a given single context through tasks such as transitive inference and graph traversal. In contrast, INSTILL investigates whether LLMs can infer underlying relational structures from natural-language descriptions across different contexts, even when entities, expressions, and domains differ.

  2. Task Construction: ReCogLab only utilizes linear chains and randomly generated tree structures. In contrast, INSTILL constructs five representative structural families motivated by cognitive science, including chains, hubs, trees, communities, and random graphs. Although chains and trees appear in both frameworks, their inclusion in INSTILL is motivated by their established roles in structural knowledge representation rather than by ReCogLab's task construction. INSTILL independently constructs these structures using structure-specific generation procedures, rather than simply adopting the construction approach used in ReCogLab.

  3. Empirical Findings: ReCogLab examines how relational reasoning performance is affected by task difficulty, context length, and premise presentation. In contrast, INSTILL provides an analysis of how structural inference varies across structure families, domain congruency, and structure sizes.

While these distinctions are discussed in Section 2, we have further emphasized the differences between the two frameworks to more clearly highlight the contributions of INSTILL (Lines 101-142).

W2. The next biggest weakness I see is that I am not really sure that the question and the way they ask it really sheds much light on the transfer of structural knowledge observed in humans. (...) It is also multiple choice rather than open ended. So is it really showing transfer of knowledge, or is it really just asking the LLM to compare graphs? (...) I was not very convinced by the comparison to human cognition and even if I was, I am not really sure what takeaways I should be taking from this paper. In general this paper's motivation is just not very clear. (...)

INSTILL investigates whether LLMs can recognize and transfer relational structures across different contexts. Importantly, models are not given explicit graphs to compare. They must infer relational structures from cognitive scenarios described in natural-language and identify the same structure despite changes in entities, expressions, and domains. This provides a controlled way to examine the abstraction and transfer of structural knowledge. As discussed in Section 3 (Lines 146-151), structural knowledge lacks a canonical representation, making it difficult to assess directly. Therefore, we adopt a multiple-choice format to evaluate this capability without requiring models to explicitly represent the inferred structures.

Cognitive science-inspired evaluations often assess latent cognitive abilities through observable behavior on controlled tasks, without assuming that the underlying mechanisms are identical to those of humans [1, 2]. For example, CogEval [1] evaluates cognitive maps and planning through tasks that vary underlying structures, without aiming to directly compare LLM performance with humans or establish that LLMs exhibit human-like cognition. ReCogLab [2] also draws on cognitive science experiments to investigate relational reasoning in LLMs, while cautioning that similarities in observed behavior do not necessarily imply identical cognitive mechanisms. In line with this perspective, INSTILL aims to characterize the ability of LLMs to generalize relational structures, rather than establish that they acquire or represent structural knowledge in the same way as humans. Our findings reveal how this ability varies across structural families, domain shifts, and structure sizes, with substantial differences in robustness across models.

[1] Momennejad et al., Evaluating Cognitive Maps and Planning in Large Language Models with CogEval, NeurIPS 2023.
[2] Liu et al., ReCogLab: A Framework Testing Relational Reasoning & Cognitive Hypotheses on LLMs, ICLR 2025.

AuthorsOpenReview ↗

Response to comments on dataset details and the reliability of LLM-generated data

W3. The details of construction are barely mentioned in the main body of the paper and do not convey a very strong sense of what this dataset really is. Section 3 is dedicated almost entirely to describing the cognitive science background of different things but not so much on details of the dataset. Like, how many nodes are in the graph, how many entities are there, how long are these sentences, what do they actually look like, how many questions is the dataset? What is the source of the data?

We clarify the data source, construction process, and characteristics of INSTILL as follows:

  1. The source of the data: As described in Section 3.3.2 (Lines 302–311), INSTILL generates synthetic tasks rather than relying on external sources. The underlying structures of reference and candidate options are constructed programmatically, while GPT-4.1 is used to generate natural-language descriptions of predefined relations.

  2. Dataset examples: Figure 2 illustrates the overall task generation process. As noted in Section 3.3.2 (Lines 302–311), additional examples are provided in Appendix A.2. Specifically, Figure 11 presents representative natural-language contexts across nine domains, while Figures 13–16 provide complete examples of generated tasks, including reference contexts and four candidate options. Also, representative samples of generated tasks are provided in our codes.

  3. The number of questions: As indicated in Section 4.1, additional experimental details are provided in Appendix B (Lines 903–917), which specifies that we generate 250 problems per structure family for the evaluations in Sections 4.2 and 4.3, and 1,000 random graph problems for the scaling analysis in Section 4.4.

We acknowledge that several descriptive statistics could be presented more explicitly in the main text. The underlying graphs range from 3 to 18 nodes and from 2 to 40 edges, with averages of 7.72 nodes and 10.13 edges per graph. Each context contains an average of 10.13 sentences, 109.2 words, 688.3 characters, and 132.4 tokens. Across all contexts, sentences contain an average of 10.87 words. To address this, we have further added the detailed statistics in Appendix B (Lines 907-910).

W4. I am kind of skeptical of LLM-generated datasets that are then used to evaluate LLMs. It seems that the dataset is a main contribution and this was mostly done by LLMs rather than collecting a set more deliberately or with human annotators. There is human validation, but there are fairly well know language and name biases in LLMs and we know that LLMs treat LLM generated language very differently from humans. So to me it seems inappropriate to consider an LLM-generated dataset an ICLR worthy submission.

We would like to clarify that our primary contribution is a framework for evaluating how LLMs infer and transfer relational structures across contexts, rather than the generated dataset itself. Importantly, the underlying graph structures and correct answers are constructed and verified programmatically, while the LLM is used only to verbalize predefined relations. We further conducted an independent human audit to verify that the generated sentences preserve the intended entities and relations.

We acknowledge that linguistic patterns and entity names in LLM-generated descriptions may introduce biases that influence model predictions. However, LLM-assisted benchmark construction has been increasingly explored to support scalable and systematic evaluation [3, 4, 5, 6]. The reliability of an LLM-generated benchmark depends not simply on the use of LLMs, but on whether the generation process is carefully controlled and the resulting data are rigorously validated to ensure their suitability for the intended evaluation. In this regard, we believe that our designed task generation process, programmatic verification, and independent human validation support the reliability of our evaluation of structural inference across different structures and domains.

[3] Jiang et al., Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testing, ICML 2025.
[4] Yuan et al., LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient, ACL 2026.
[5] Huang et al., DataGen: Unified Synthetic Dataset Generation via Large Language Models, ICLR 2025.
[6] Li et al., AutoBencher: Towards Declarative Benchmark Construction, ICLR 2025.

AuthorsOpenReview ↗

Response to comments on task design and details on the construction of the dataset

W5. For experiments, first, from the node ablation in Figure 6 it really does not seem like these graphs are very large at all. And looking at the prompt, I am also wondering how much work was done on this prompt because at least to me, "underlying structure" seems very vague and if you didn't tell me I might not actually know what it was asking either. Is it possible that with some prompt tuning or in-context examples, more of these models would do better?

While the graphs used in INSTILL are relatively small, our objective is not to evaluate performance on large graphs, but to examine how LLMs infer and transfer relational structures across contexts. Even with these relatively small graphs, our results reveal that structural inference varies across structure families and domain incongruency across diverse model families. Figure 6 further examines how performance changes as the number of edges increases within the evaluated range.

We would like to clarify that the term “underlying structure” follows its use in prior cognitive science research on structural knowledge as discussed in the related work [1, 7, 8]. In INSTILL, it specifically refers to the pattern of relationships among entities, including relation direction where applicable. As described in Section 3.1, the instruction explicitly refers to a “graph-like structure” to clarify that models should compare relational patterns across contexts rather than surface-level similarities.

We agree that prompt tuning and in-context demonstrations may improve accuracy. However, our primary objective is to investigate LLMs’ ability to infer and transfer relational structures, rather than to maximize their performance through task-specific prompting. Exploring how different prompting strategies influence structural inference remains an interesting direction for future work.

[7] Kemp & Tenenbaum, The discovery of structural form, PNAS 2008.
[8] Mark et al., Transferring structural knowledge across cognitive maps in humans and models, Nature Communications 2020.

Q1. Can you please give more details on the construction of the dataset including the ones I listed above?

As described in Section 3, we first generate graphs from five structural families, with nodes representing entities and edges representing relations. These graphs contain 3 to 18 nodes, corresponding to the same number of entities. Each edge is then expressed as a natural-language sentence using GPT-4.1. The resulting cognitive scenarios span nine domains, and each question consists of a reference context and four candidates, exactly one of which preserves the reference structure. The correct answers are verified programmatically. Appendix A also provides the detailed generation procedure and representative examples, including complete questions.

As specified in Appendix B, we generate 250 questions per structure family for the evaluations in Sections 4.2 and 4.3, and 1,000 random graph questions for the scaling analysis in Section 4.4. Figure 6 further illustrates the edge counts considered in the scaling analysis.

Reviewer_v2SqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces INSTILL, a framework for evaluating whether LLMs can identify matching relational structures across natural-language contexts. It covers five structural families and nine domains, with experiments on domain shifts and structure size. The results show strong performance from several recent models and greater sensitivity to domain cues in weaker models.

Critical strengths
  1. The framework provides a controlled way to study structural matching across different domains. Programmatic answer verification and the human audit strengthen confidence in the generated tasks.

  2. Particularly, the congruent and incongruent conditions are informative: they help distinguish successful cross-domain matching from reliance on semantic similarity.

Critical weaknesses
  1. The prompt tells the model directly to look for identical graph structure. This is a reasonable starting point, but I would like to know whether the results hold when the task is phrased without this explicit cue. Given prior findings on prompt sensitivity [1], a small comparison using clear, application-based instructions would help establish how broadly the findings apply.

  2. Models might still identify the answer from a few distinctive features, such as a prominent hub or local connection patterns. Prior work has found biases toward particular local structures in graph recall [2]. Adding harder distractors with the same node count, edge count, and degree sequence would make the evidence for global structural matching more convincing.

[1] Sclar et al. Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I Learned to Start Worrying About Prompt Formatting. ICLR, 2024.

[2] Wang, Cui, and Kleinberg. Microstructures and Accuracy of Graph Recall by Large Language Models. NeurIPS, 2024.

Questions for authors
  1. Does accuracy remain stable when explicit references to graphs and structural identity are replaced with an equally clear application-based task requiring the same matching decision?

  2. Can you evaluate a subset with non-isomorphic distractors that share the reference's node count, edge count, and degree sequence, to test whether global connectivity is necessary?

1 comment
AuthorsOpenReview ↗

Response to comments on alternative task instructions and degree sequence controls

We sincerely appreciate your careful evaluation and valuable feedback on our manuscript. We hope that our responses to your comments below clearly address your concerns.

W1. The prompt tells the model directly to look for identical graph structure. (...) Given prior findings on prompt sensitivity [1], a small comparison using clear, application-based instructions would help establish how broadly the findings apply.

Q1. Does accuracy remain stable when explicit references to graphs and structural identity are replaced with an equally clear application-based task requiring the same matching decision?

INSTILL evaluates whether LLMs can infer the relational structures within a cognitive scenario described in natural-language and transfer them across diverse contexts. The original instruction explicitly directs models to compare relational structures, but identifying the correct candidate still requires inferring these structures from natural language. However, a generic instruction to identify a matching context without specifying the basis for comparison could allow models to rely on lexical or semantic similarity rather than relational structure. In such cases, it would be difficult to disentangle whether performance degradation stems from limitations in structural inference or ambiguity in the task instruction. For this reason, we explicitly specify the basis for comparison in the original instruction to ensure that models are evaluated under a consistent criterion.

However, following the reviewer's suggestion, we conducted an additional experiment with the GPT series, replacing the original instruction, “Select the option whose underlying graph-like structure is structurally identical,” with “Which situation preserves the same relationships as the reference?” without explicitly referring to graphs or structures. The results show that the performance trends across models remain largely unchanged, consistent with the findings of our original evaluation. Although accuracy decreases overall, the extent of the decline varies across models, potentially reflecting differences in their sensitivity to less explicit task instructions. Notably, GPT-5.4 and GPT-5 exhibit relatively small performance declines, whereas their mini variants show larger declines, consistent with the performance trends observed in our original evaluation. We have added experimental results and further discussion in Appendix C.5.

W2. Models might still identify the answer from a few distinctive features, such as a prominent hub or local connection patterns. (...) Adding harder distractors with the same node count, edge count, and degree sequence would make the evidence for global structural matching more convincing.

Q2. Can you evaluate a subset with non-isomorphic distractors that share the reference's node count, edge count, and degree sequence, to test whether global connectivity is necessary?

We would like to clarify that, as described in Section 3.2 (Lines 175-183) and Appendix C.1.1 (Lines 985-992), our task construction controls for graph size. Specifically, the reference and all candidate structures have the same number of nodes in every structure family. They also share the same number of edges, except for community structures, where the number of edges depends on the number and sizes of communities. These settings rule out graph size as a shortcut for identifying the correct candidate within each question. We have further clarified this construction procedure in the manuscript by explicitly stating how node and edge counts are matched across contexts. (Lines 181-183)

We agree that constructing a subset with non-isomorphic distractors sharing the reference’s degree sequence would provide a stricter test. Matching degree sequences would eliminate simple degree-based cues, such as the presence of a prominent hub, and allow us to assess whether models can distinguish different connectivity patterns under this constraint.

Following the reviewer’s suggestion, we conducted an additional experiment with the GPT series using questions from the random graph family. Within each question, we fixed the domain, node count, and edge count across the reference and all candidates. We then compared two conditions: one where all candidates shared the reference’s degree sequence (matched) and another where only the correct candidate did (unmatched). In both conditions, the distractors were non-isomorphic to the reference. GPT-5.4 and GPT-5 maintained high accuracy even when degree sequences were matched, whereas their mini variants, which were also more susceptible to distractors sharing the reference domain in Table 2, exhibited larger performance declines. This pattern is consistent with our main findings, as models that achieve higher performance on structural inference tasks tend to show smaller performance declines under more challenging conditions. We have added experimental results and further discussion in Appendix C.6.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.