- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper targets a known limitation of conventional cognitive diagnosis models: they model exercises only at the coarse-grained concept level and cannot distinguish heterogeneous reasoning demands within the same concept, such as two SQL exercises both tagged JOIN that may require either simple recall or multi-step schema inspection and query construction. The paper proposes the task of Cognitive Route Induction, or CRI, together with a diagnosis model named ROUTECD. During training, multi-round LLM consistency voting generates weak supervision that guides a graph neural network to parse exercise text into directed cognitive route graphs grounded in Bloom's taxonomy. ROUTECD then maps exercise demands and student historical states into a shared concept-by-process space and predicts responses from the cognitive gap between them, requiring zero LLM calls at inference.
Critical strengths
Extending the conventional one-dimensional notion of concept mastery to a two-dimensional concept-by-process matrix, and modeling exercise prerequisite dependencies as a non-chain directed graph, is a well-motivated direction with good educational-measurement intuition. The design of distilling LLM signals at training time while remaining LLM-free at inference also addresses practical deployment concerns regarding latency.
The paper provides extensive baseline comparisons on four datasets, including recent baselines such as CCD and EduEmbed, with supplementary analyses covering multi-LLM teacher consistency, human expert blind evaluation, and cold-start settings. Source code is released in an anonymous repository, providing a reasonable basis for reproducibility.
The symbol definitions, the derivations, and the prompt templates in the appendix are fairly detailed, though the extremely poor rendering of the figures severely undermines overall readability.
Critical weaknesses
The core architecture and result figures are too low-resolution to read. The flagship Figure 1 on page 1 and the overall architecture in Figure 2 on page 2 are heavily compressed and pixelated: the SQL question text, small annotations, and matrix labels are illegible. Likewise, appendix Figures 6 through 10 and Table 3, which is pasted as a low-resolution bitmap screenshot rather than a native LaTeX table, are blurry, with axis labels and bar-chart numbers nearly unreadable. Such careless visual presentation does not meet the acceptance standard of ICLR.
The predicted directed edges are disconnected from diagnosis. A core selling point is inducing directed prerequisite edges through Equation 9, supervised by the edge loss and the DAG regularization loss. However, a close reading of Equations 4 through 10 reveals that message passing uses Softmax weights computed from layer-0 features in Equation 5, while the supervised edge predictions, which are a layer-T output per Equation 9, are used neither in message passing nor in the node pooling of Equation 10, which only weights the final node states by activation probabilities. Consequently, at inference, the predicted directed edges contribute nothing along any forward-computation path to the final prediction, degenerating into a purely auxiliary training branch.
The ablation and hyperparameter analyses contradict each other. In Table 3, removing LLM supervision entirely, that is the Text plus CRI variant, achieves an AUC of 0.8095, essentially identical to the full ROUTECD result of 0.8099, a difference of only 0.0004. Yet in Section 4.4 around Line 482 and in Figure 8b, the authors claim that removing LLM supervision by setting the node supervision coefficient to zero causes AUC to drop to 0.7969. These two findings on the same dataset DBE-KT22 are in direct conflict.
Questions for authors
Why were Figures 1, 2, and 6 through 10 as well as Table 3 submitted as such low-resolution bitmaps? The authors must replace all of them with vector graphics and native LaTeX tables in any revision.
The directed edges predicted in Equation 9 never enter the computation of the pooled route representation in Equation 10, since message passing only uses the layer-0 coefficients. If the edge predictor is simply discarded at inference, does the final prediction change at all? If not, how can the induced edges be claimed as functional diagnostic structures per Line 96 rather than a purely auxiliary training-time regularizer?
Please clarify why the Text plus CRI variant, with no LLM supervision whatsoever, reaches an AUC of 0.8095 in Table 3, whereas removing LLM supervision in Section 4.4 and Figure 8b yields 0.7969. If the Text plus CRI variant achieves 0.8095 without any LLM supervision, what exactly is the practical contribution of LLM distillation to response prediction?
Why does forcing the demand matrix to align with the true knowledge concepts through hard or soft masks cause AUC to drop by nearly three points, falling below the base backbone? Moreover, how much performance remains if the bypass of the raw text and student representations in Equation 19 is removed and a non-negativity constraint enforcing monotonicity is added?
Dear Reviewers,
Please note that the author rebuttal is now available. We kindly ask you to read it, join the discussion, and update the review as soon as possible. Your prompt action will help us proceed to the final decision. Thanks.
Best regards
AC