acceptodds
Withdrawn from ICLR 2027

ReTraceKT : Relevance-guided Multimodel State-space Knowledge Tracing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Knowledge tracking (KT) estimates a learner's evolving knowledge state from a sequence of exercise responses. Two properties of modern learning platforms make this problem harder: essential evidence is often embedded in diagrams rather than text, and long histories contain many interactions that are irrelevant to the next exercise. We present ReTraceKT, a multimodal KT framework that retrieves semantically relevant evidence before tracking knowledge dynamics. ReTraceKT first uses a vision-language model offline to render each exercise image into structured descriptions and aligns them with trainable exercise embeddings. It then builds a threshold semantic graph over exercises and uses the graph to retrieve a sparse, question-conditioned summary of a learner's history. Finally, a selective state-space tracer combines local response patterns with long-range state transitions in linear sequence complexity. Experiments on NIPS34, MOOC-Radar, and XES3G5M show consistent improvements over strong recurrent, attention-based, and sparse KT baselines. Relative to the strongest baseline, ReTraceKT improves AUC by 3.22%, 1.82%, and 2.08% on the three datasets, respectively; ablations confirm that semantic rendering and relevance retrieval are complementary. Overall, ReTraceKT delivers consistently strong and stable performance across the three datasets.

Primary area:
learning on time series and dynamical systems
Keywords:
knowledge tracing, multimodal representation, state-space models, relevance retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_faYGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a framework for multimodal knowledge tracing. It uses a vision–language model to convert exercise images into structured descriptions offline. These descriptions are encoded to represent exercises and build a thresholded semantic graph. It then retrieves relevant past interactions and processes them with a selective state-space model. The authors report better performance than recurrent, attention-based, and sparse KT baselines on NIPS34, MOOC-Radar, and XES3G5M. They also provide ablation studies, sensitivity analyses, runtime comparisons, and qualitative visualizations to further assess the method.

Critical strengths
  1. This paper jointly addresses visual exercise content and irrelevant historical interactions, which makes the proposed direction useful for educational platforms.

  2. Potentially practical deployment design. Separating offline content understanding from online tracing allows expensive VLM outputs to be reused across learners, which potentially reduces inference overhead.

Critical weaknesses
  1. Although selective state-space modeling is central to the framework, Table 2 omits Mamba4KT, which the paper itself cites. There is also no controlled comparison replacing the SSM with an LSTM or Transformer while retaining the same semantic inputs and retrieval module. Consequently, the experiments do not adequately distinguish gains from the proposed retrieval mechanism, additional pretrained semantic information, and the backbone architecture.

  2. Removing structured semantic rendering reduces the model to ID/concept embeddings, eliminating content information altogether. This demonstrates the potential value of additional content, but does not establish that the proposed three-stage rendering is superior to OCR text, original exercise text, or direct image features. The paper also does not assess the accuracy of generated visual relations or performance specifically on exercises requiring visual information.

  3. Equation (5) defines vqv_q using field projections WkW_k, while lines 172–174 state that all vqv_q are computed once and cached, and that the field projections are optimized with KT supervision. Updating these projections would generally change both vqv_q and the graph derived from it. Conversely, keeping the final representations fixed would prevent downstream gradients from updating the projections through that path. The paper needs to specify what is cached, when the graph is updated, and how gradients propagate. Without these details, the training procedure is not unambiguously defined.

  4. A linear-time SSM scan does not imply linear complexity for the full model, which also performs history retrieval. A fixed similarity threshold does not bound the number of retained neighbors. When historical exercises are highly similar, the operator in Equation (8) can still contain O(L2)O(L^2) nonzero entries. The paper does not report how retained-edge counts scale with sequence length, and Figure 7 lacks sufficient information about hardware, batch size, and the scope of timing. These omissions limit the claimed efficiency advantage.

Questions for authors
  1. Table 2 contains mathematically inconsistent probability-error metrics.

    For LefoKT on MOOC-Radar, the reported RMSE and MAE are 0.38900.3890 and 0.14830.1483, respectively. For binary labels and probability predictions, ei=∣pi−yi∣∈[0,1]e_i=|p_i-y_i|\in[0,1], so ei2≤eie_i^2\le e_i. Therefore,

    RMSE2≤MAE.\mathrm{RMSE}^2\le \mathrm{MAE}.

    However,

    0.38902=0.151321>0.1483.0.3890^2=0.151321>0.1483.

    These values cannot arise from the same probability predictions evaluated over the same samples with consistent weighting.

  2. The matrix products in Equations (8) and (14) are dimensionally invalid as written.

The paper defines Z∈RL×dZ\in\mathbb R^{L\times d} and G∈RL×LG\in\mathbb R^{L\times L}, while Equation (7) implies a feature projection Wg∈Rd×dW_g\in\mathbb R^{d\times d}. Thus, Equation (8),

Z~=Z+GWgZ,\widetilde Z=Z+GW_gZ,

is not a valid matrix expression for general L≠dL\ne d. A dimensionally consistent formulation matching the per-position feature projection would be

Z~=Z+GZWg⊤.\widetilde Z=Z+GZW_g^\top.

Equation (14) repeats the same ordering problem.

Reviewer_5XNzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address the difficulties existing KT methods face in handling figures / graphs associated with questions and finding the most relevant information from long learner histories. The authors present ReTraceKT, a multimodal KT framework that uses a VLM to summarize exercise images into structured records containing question text, answer options, and visual information, then retrieves relevant historical interactions based on question semantic similarity, and finally uses a selective state-space model to track changes in learner states and predict future responses. They evaluate the framework on NIPS34, XES3G5M, and MOOC-Radar against several existing KT methods and provide ablation results on multiple aspects.

Critical strengths
  1. The two problems the paper aims to address are meaningful and important for KT. Using visual information in exercises and identifying relevant interactions in long learner histories could help models make better use of the information available on learning platforms.

  2. Combining relevance-guided retrieval with an SSM is a reasonable design choice. Retrieval provides access to relevant past interactions, while the SSM processes temporal information efficiently. This combination offers a promising approach to modeling long learner histories.

Critical weaknesses
  1. Missing details and inconsistent descriptions make the method difficult to understand precisely. Some computational steps in Figure 2 do not match the equations, and implementation details are missing or unclear. This makes it difficult to determine the actual method used in the experiments. I list my specific concerns in the Questions section below.

  2. It is unclear whether the improvements mainly come from richer exercise content or from the proposed architecture. The paper does not specify which VLM is used or what content information each baseline receives. Removing SR substantially reduces performance, but it also removes the additional semantic information. This ablation therefore does not clarify whether the gains mainly from the access to richer content.

Questions for authors
  1. In Figure 2, Path B follows Conv → SSM → SiLU, while Eq. (9) applies SiLU after the convolution and before the SSM in Eqs. (10)–(13). Which computation matches the implementation? Also, does “Layer-wise Filtering” in Path A correspond to the retrieval operation in Eq. (14)? If so, the figure places it in the gating branch, while Eq. (14) combines retrieved history with the previous layer’s representations before feeding the result into the SSM in Eq. (15). Please clarify this difference, or explain which operation the filtering block represents.

  2. What is D in Eq. (13), and is it a learned parameter or a fixed mapping? Also, Eq. (5) computes v_q using field projections W_k, which the text describes as trainable, while stating that all v_q are computed once and cached. What exactly is cached?

  3. Which specific VLM and TRM are used?

  4. Does the current comparison include any strong baseline that can read multimodal exercise content? If not, I suggest adding a suitable comparison, either with an existing multimodal KT method or by providing the cached semantic features ReTraceKT used to a context-aware KT baseline.

  5. What proportion of exercises and interactions in each evaluation dataset involve diagrams? Since removing SR also removes question text and answer options, a variant that retains these two fields and removes only the visual-relations description would help distinguish the benefit of diagram descriptions from that of access to basic question content.

Reviewer_H7yrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ReTraceKT, a multimodal knowledge tracing (KT) framework addressing limitations in exercise representation and long-history modeling. It combines three components: (1) offline VLM-based rendering of exercise text, answer options, and visual relations into semantic representations; (2) relevance-guided aggregation of historical interactions using an exercise-similarity graph with threshold τ = 0.2; and (3) a selective state-space model (SSM) for tracking student knowledge dynamics. Experiments on NIPS34, MOOC-Radar, and XES3G5M compare ReTraceKT against 13 baselines, including DKT, DKVMN, SAKT, AKT, RKT, SAINT, SparseKT, ExtraKT, MIKT, HD-KT, LefoKT, RobustKT, and KeenKT. The reported AUC scores are 0.8145, 0.8658, and 0.8082, outperforming the strongest reported baselines by 3.22%, 1.82%, and 2.08% relatively. Ablations suggest that semantic rendering contributes more than relevance retrieval, while runtime comparisons indicate improved efficiency over several attention-based methods. The authors conclude that integrating multimodal semantics, sparse retrieval, and state-space modeling improves KT accuracy and efficiency.

Critical strengths
  1. Relevant problem formulation and potentially useful architecture. The paper addresses two meaningful challenges in knowledge tracing: extracting information from multimodal exercises and identifying relevant historical interactions. Exercise images can contain information that is not adequately represented by question IDs or knowledge-concept labels. Using a VLM to extract structured exercise semantics is a reasonable approach to addressing this limitation. Separating offline content understanding from online student-state modeling is a reasonable and practically motivated strategy for reducing repeated VLM inference costs.

  2. An attempt to jointly evaluate predictive performance and computational efficiency. The paper evaluates the proposed framework across three datasets and includes component ablations, sequence-length sensitivity analyses, retrieval visualizations, and runtime comparisons. This is a very useful and well-established evaluation scope for a system combining semantic retrieval with sequence modeling. However, the value of these results depends on resolving the implementation and evaluation concerns described below.

Critical weaknesses
  1. Potential causal label leakage and inconsistencies between the paper and code. Eq. 7–8 explicitly require causal retrieval, but the released SimilarityAttention constructs a full similarity matrix without causal masking. Since the input sequences contain shifted responses, predictions at position (i) can aggregate representations from future positions containing the target response (r_{i+1}). This aggregation occurs both before and after the Mamba blocks. Diagnostic tests further support the leakage concern: perturbing future responses changes earlier predictions, while a synthetic random-label experiment reportedly achieves 0.770 AUC with the original implementation versus 0.499 after causal masking. Although the effect on real datasets remains unknown, this raises serious concerns about the reported results. Additional discrepancies include missing semantic fusion and prediction components described in Eq. 3–5 and Eq. 17, inconsistent hyperparameters, and unclear separation of validation and test data. If the reported results used this implementation, they may be contaminated; otherwise, the released code does not reproduce the described method.

  2. Questionable baseline comparisons and insufficient evaluation reliability. Several Table 2 baselines substantially underperform published benchmarks. On XES3G5M, the reported AUCs for AKT, SAINT, and SparseKT are 0.7902, 0.7627, and 0.7263, compared with published values of 0.8207, 0.8074, and 0.8165. ReTraceKT's 0.8082 is also below these published results. Similar discrepancies occur on NIPS34, including KeenKT (0.7891 versus its originally reported 0.8162). Although cross-paper comparisons may involve different splits, the near-matching DKT/DKVMN results alongside substantially weaker question-aware baselines raise concerns about input-feature fairness. Furthermore, the released code does not establish reproducible shared splits or an independent test evaluation, and important baselines such as Mamba4KT, simpleKT, and content-enhanced AKT are missing. Consequently, the claimed improvements remain insufficiently supported.

Questions for authors

Q1. Causal correctness (critical clarification) Can the authors provide results from a corrected implementation with causal masking and demonstrate that future response perturbations do not alter earlier predictions?

Q2. Validity of the reported results (critical clarification) Were the reported AUC values obtained using an independent test set, or are they best-validation scores selected through early stopping? Please clarify how student-level splits were generated, whether they were shared across methods (seeds?), and how run-to-run variability was measured. A complete rerun with fixed splits, an independent test set, and reported standard deviations is necessary.

Q3. Baseline fairness (critical clarification) Why do several attention-based baselines substantially underperform previously published benchmark results, while DKT and DKVMN remain relatively close? Did AKT, SAINT, and sparseKT receive the same question IDs, concept IDs, and available auxiliary information as ReTraceKT? Can the authors provide complete baseline configurations and comparisons with Mamba4KT, simpleKT, and content-enhanced AKT?

Q4. Implementation consistency (critical clarification) Which retrieval mechanism was actually evaluated: the historical-query formulation in Eq. 7–8 or the next-question-based indexing in the code? It is different between paper and code. Can the authors reconcile the missing field-wise semantic fusion and prediction MLP, as well as the differences in state size, sequence length, optimizer configuration, and retrieval order?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.