acceptodds
Under review as a conference paper at ICLR 2027

One Size Fits None: Exercise-Adaptive Cognitive Route Induction for Cognitive Diagnosis with LLM Supervision

Abstract

Cognitive diagnosis aims to identify students' learning weaknesses, yet most existing models operate at the concept level, representing exercises as item IDs, Q-matrix entries, concept labels, or dense embeddings. Such representations obscure heterogeneous reasoning demands within the same concept: SQL exercises tagged with JOIN may require either simple recall or complex schema inspection, key-relation reasoning, constraint filtering, and query construction. To address this limitation, we propose Cognitive Route Induction (CRI), which parses exercise text into a Bloom-based cognitive route graph, where nodes denote activated cognitive processes and edges encode prerequisite dependencies. CRI defines the target representation for fine-grained diagnosis: the cognitive operations required by an exercise and their dependencies. Building on CRI, we introduce RouteCD, a route-aware cognitive diagnosis model that projects each exercise route into a shared concept-by-process space and predicts responses from the gap between student cognitive state and exercise demand. LLM-generated analyses provide node- and edge-level graph supervision during training, while inference requires no LLM calls. Experiments show improved response prediction and cognitive-structure modeling, and ablations confirm the diagnostic value of induced routes. Our code is available at: https://anonymous.4open.science/r/ROUTECD.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Cognitive Diagnosis, Exercise Understanding, Text-to-Graph Learning, Bloom's Taxonomy, LLM Supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fsu8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper targets a known limitation of conventional cognitive diagnosis models: they model exercises only at the coarse-grained concept level and cannot distinguish heterogeneous reasoning demands within the same concept, such as two SQL exercises both tagged JOIN that may require either simple recall or multi-step schema inspection and query construction. The paper proposes the task of Cognitive Route Induction, or CRI, together with a diagnosis model named ROUTECD. During training, multi-round LLM consistency voting generates weak supervision that guides a graph neural network to parse exercise text into directed cognitive route graphs grounded in Bloom's taxonomy. ROUTECD then maps exercise demands and student historical states into a shared concept-by-process space and predicts responses from the cognitive gap between them, requiring zero LLM calls at inference.

Critical strengths

Extending the conventional one-dimensional notion of concept mastery to a two-dimensional concept-by-process matrix, and modeling exercise prerequisite dependencies as a non-chain directed graph, is a well-motivated direction with good educational-measurement intuition. The design of distilling LLM signals at training time while remaining LLM-free at inference also addresses practical deployment concerns regarding latency.

The paper provides extensive baseline comparisons on four datasets, including recent baselines such as CCD and EduEmbed, with supplementary analyses covering multi-LLM teacher consistency, human expert blind evaluation, and cold-start settings. Source code is released in an anonymous repository, providing a reasonable basis for reproducibility.

The symbol definitions, the derivations, and the prompt templates in the appendix are fairly detailed, though the extremely poor rendering of the figures severely undermines overall readability.

Critical weaknesses

The core architecture and result figures are too low-resolution to read. The flagship Figure 1 on page 1 and the overall architecture in Figure 2 on page 2 are heavily compressed and pixelated: the SQL question text, small annotations, and matrix labels are illegible. Likewise, appendix Figures 6 through 10 and Table 3, which is pasted as a low-resolution bitmap screenshot rather than a native LaTeX table, are blurry, with axis labels and bar-chart numbers nearly unreadable. Such careless visual presentation does not meet the acceptance standard of ICLR.

The predicted directed edges are disconnected from diagnosis. A core selling point is inducing directed prerequisite edges through Equation 9, supervised by the edge loss and the DAG regularization loss. However, a close reading of Equations 4 through 10 reveals that message passing uses Softmax weights computed from layer-0 features in Equation 5, while the supervised edge predictions, which are a layer-T output per Equation 9, are used neither in message passing nor in the node pooling of Equation 10, which only weights the final node states by activation probabilities. Consequently, at inference, the predicted directed edges contribute nothing along any forward-computation path to the final prediction, degenerating into a purely auxiliary training branch.

The ablation and hyperparameter analyses contradict each other. In Table 3, removing LLM supervision entirely, that is the Text plus CRI variant, achieves an AUC of 0.8095, essentially identical to the full ROUTECD result of 0.8099, a difference of only 0.0004. Yet in Section 4.4 around Line 482 and in Figure 8b, the authors claim that removing LLM supervision by setting the node supervision coefficient to zero causes AUC to drop to 0.7969. These two findings on the same dataset DBE-KT22 are in direct conflict.

Questions for authors

Why were Figures 1, 2, and 6 through 10 as well as Table 3 submitted as such low-resolution bitmaps? The authors must replace all of them with vector graphics and native LaTeX tables in any revision.

The directed edges predicted in Equation 9 never enter the computation of the pooled route representation in Equation 10, since message passing only uses the layer-0 coefficients. If the edge predictor is simply discarded at inference, does the final prediction change at all? If not, how can the induced edges be claimed as functional diagnostic structures per Line 96 rather than a purely auxiliary training-time regularizer?

Please clarify why the Text plus CRI variant, with no LLM supervision whatsoever, reaches an AUC of 0.8095 in Table 3, whereas removing LLM supervision in Section 4.4 and Figure 8b yields 0.7969. If the Text plus CRI variant achieves 0.8095 without any LLM supervision, what exactly is the practical contribution of LLM distillation to response prediction?

Why does forcing the demand matrix to align with the true knowledge concepts through hard or soft masks cause AUC to drop by nearly three points, falling below the base backbone? Moreover, how much performance remains if the bypass of the raw text and student representations in Equation 19 is removed and a non-negativity constraint enforcing monotonicity is added?

Reviewer_N58FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses heterogeneous cognitive demands among exercises associated with the same knowledge concept. It introduces Cognitive Route Induction (CRI), which maps exercise text and concept metadata into Bloom-based cognitive route graphs, with nodes representing cognitive processes and directed edges encoding prerequisite dependencies. Building on CRI, ROUTECD represents exercise demand and student state in a shared concept-by-process space, using their cognitive gap to predict responses and identify potential weaknesses. LLM-generated analyses provide node- and edge-level supervision during training, while inference requires no LLM calls. Evaluation on four educational datasets includes predictive comparisons, component ablations, sensitivity to LLM teachers and text encoders, cold-start settings, cross-domain transfer, and expert assessment of cognitive routes. The results indicate benefits, although their magnitude and statistical significance vary across datasets.

Critical strengths
  1. The paper addresses an educationally meaningful problem with a richer diagnostic representation. Exercises associated with the same concept can impose substantially different cognitive demands. CRI and ROUTECD seek to move diagnosis from identifying weak concepts toward identifying weak cognitive processes within those concepts, using cognitive routes and a shared concept-by-process representation. This offers a valuable research direction beyond coarse concept-level diagnosis and is a primary consideration supporting acceptance.

  2. The evaluation is broad and transparent about the method’s limitations. The experiments cover component ablations, different LLM teachers and text encoders, cold-start settings, cross-domain transfer, and expert assessment of cognitive routes. The appendix also reports nonsignificant improvements in some settings, remaining cyclic routes, and demand assigned to unannotated concepts. This evidence helps assess both the benefits and boundaries of the approach, strengthening the paper’s credibility.

Critical weaknesses
  1. The cognitive route representation is too coarse to substantiate step-level diagnosis. CRI assigns one node to each Bloom process, merging operations involving different content. For example, “understand key relationships → apply JOIN” and “understand business aggregation criteria → evaluate query results” share a single Understand node, losing the explicit correspondence between individual operations and their downstream dependencies. The graph therefore captures aggregated dependencies between cognitive categories, and its ability to support step-level reasoning-breakdown diagnosis remains insufficiently established.

  2. Diagnostic reliability under incomplete concept annotations remains insufficiently established. Concept metadata informs route induction, but exercise annotations may omit relevant knowledge requirements. Although the model can assign demand to unannotated concepts within the predefined inventory, Table 15 does not establish whether such assignments recover meaningful dependencies or reflect spurious allocation. Concepts outside the inventory also lack explicit diagnostic dimensions. These limitations leave uncertainty about whether errors caused by missing knowledge requirements are correctly attributed.

Questions for authors

Clarification: What is the primary benefit of LLM-to-graph supervision? Table 9 shows no statistically significant differences between ROUTECD and the strongest metric-wise baselines on ASSIST and DBE-KT22. Separately, Table 3 shows that removing LLM graph supervision reduces DBE-KT22 AUC only from 0.8099 to 0.8095, although RMSE worsens from 0.3740 to 0.3787. Does this supervision primarily improve ranking performance, probability error, or cognitive-structure quality, and which benefits have statistical support? This would affect my assessment of its necessity and contribution.

Reviewer_kRAkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work targets fine-grained cognitive diagnosis for student assessment. The authors propose Cognitive Route Induction (CRI) to convert exercises into Bloom-based cognitive route graphs capturing cognitive operations and prerequisite dependencies, and build RouteCD, a diagnosis model mapping these routes into a concept-process space to predict student responses. LLMs supply graph supervision for training, while inference does not need LLMs. Experiments report better response prediction and cognitive structure modeling, with ablation studies validating the induced routes.

Critical strengths

1.This work moves cognitive diagnosis from coarse concept-level modeling to fine-grained cognitive-process modeling by proposing Cognitive Route Induction, which converts exercise texts into auditable Bloom-based cognitive route graphs.

2.It develops RouteCD, a route-aware cognitive diagnosis model. The model leverages induced cognitive routes to represent exercise requirements and estimates student performance by measuring the cognitive gap between exercise demand and student state, supporting both response prediction and reasoning breakdown diagnosis.

3.The work distills LLM-generated cognitive annotations into node and edge graph supervision. This strategy obtains structured cognitive signals during training and eliminates LLM inference overhead at test time.

Critical weaknesses

1.The paper claims that the gap between exercise-level cognitive demands and students’ cognitive states can identify weaknesses in specific cognitive processes. However, the evaluation focuses primarily on response prediction and the plausibility of exercise-level cognitive routes, without independent ground-truth annotations of students’ actual error causes to validate the model’s diagnostic claims.

2.The ablation studies do not sufficiently demonstrate that the observed performance gains depend on the semantic meaning and directionality of the induced cognitive routes. In Eq.(19), the prediction head directly incorporates exercise representations and student history representations, potentially allowing the model to make accurate predictions without substantially relying on the cognitive gap. This design weakens the claim that cognitive routes serve as the core diagnostic mechanism.

3.The performance gains are inconsistent across the four datasets, and the caption of Figure 3 claims that the proposed model achieves the lowest RMSE on every dataset, which is inconsistent with the results reported in Table 2. Furthermore, Appendix Table 9 shows that only 5 of the 12 comparisons against the strongest baselines across the evaluated metrics remain statistically significant after multiple-comparison correction, with some differences being numerically small.

4.Section 3.4 presents only the overall training objective, while key methodological details—including how multi-round LLM annotations are aggregated into node- and edge-level supervision, how supervision weights are determined, and how sparsity and acyclicity constraints are imposed on the predicted graphs—can only be understood by consulting the appendix.

Questions for authors

1.Do the authors have independent ground-truth annotations of students’ real error causes to verify if the model identifies specific cognitive weaknesses instead of just predicting answers?

2.Can the authors conduct controlled experiments including randomized graphs, reversed edges, undirected graphs and removed direct prediction pathways to verify whether cognitive route semantics and directionality independently boost performance?

3.How do the authors resolve the inconsistency between Figure 3 caption and Table 2, and explain the limited statistically significant results after correction? Please also report random-seed variability or interval estimates for main results.

4.How are multi-round LLM annotations aggregated into graph supervision, how are supervision weights determined, and how are sparsity and acyclicity constraints applied to predicted graphs?

Comments on the paper

1 comment

Dear Reviewers,

Please note that the author rebuttal is now available. We kindly ask you to read it, join the discussion, and update the review as soon as possible. Your prompt action will help us proceed to the final decision. Thanks.

Best regards

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.