acceptodds
Under review as a conference paper at ICLR 2027

Closing the Loop: Auditable Half-Life Prediction with Open-Space Retrieval and Capacity-Aware Scheduling

Abstract

Spaced repetition requires accurate memory-state prediction and effective scheduling under limited review capacity. We present a closed-loop framework that couples Open-Space Retrieval for Half-Life Regression (OSR-HLR) with the OSR Adaptive Calibration-Regret Index (OSR-ACRI). OSR-HLR combines relative half-life estimation with open-space retrieval, which transfers deterministic, source-traceable residual evidence from recurring and related historical memory states, while OSR-ACRI converts half-life estimates into capacity-aware review decisions using online calibration, marginal delay cost, and queue age. Across MaiMemo, Anki, and EdNet-Recall, under source-only protocols that keep every Test outcome out of training and retrieval, OSR-HLR achieves the lowest H-SMAPE among 27 baselines in all 15 dataset-protocol settings. It reduces H-SMAPE by 17.40%, 2.59%, and 42.46%, respectively, under the Row protocol, and by 11.36% on held-out Anki users. Under a fixed scheduler, improved predictions reduce 365-day regret by 8.7%-15.8%. With the predictor fixed and workload matched, OSR-ACRI improves retention AUC by 11.1%, reduces time below target by 7.5%, and improves final recall by 19.8%. Together, the two components provide an auditable prediction-to-scheduling loop for efficient spaced repetition.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Spaced Repetition, Memory Half-Life Prediction, Memory State Estimation, Review Scheduling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_SBfBOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

As a researcher in AI, this work is very difficult for me to understand. I would not attribute this case to heavy AI writing, but because very strong background mismatch.

In a more natural presentation: this work tries to optimize daily learning schedule for a human learner. To do this, one need to fit a forgetting curve (HLR), and then predict a better schedule (ACRI) given the forgetting dynamics.

I am not sure whether OSR is a standard word for education, but definitely not a common technical term in AI.

Critical strengths

This work has extensive baselines for comparing the accuracy of different algorithms.

Critical weaknesses

First, I am not sure who the intended audience is, but I am definitely not one of them. From the perspective as an AI researcher, the method design in section 4 is very specific to the dataset, and quite arbitrary. There is no scientific insight or generalizable technique.

Second, it seems half of the claim is built around the ACRI method, which is unverified in practice, but only simulated for comparison. The simulation uses the ground truth model and hence the improvement is expected.

Questions for authors

What is the key contribution from the perspective of AI researchers?

5 comments
AuthorsOpenReview ↗

Response Part 1/2

We sincerely thank the reviewer for their careful evaluation, constructive suggestions, and recognition of the strengths of our work. Below, we address your concerns point by point.

W1 / Q1 — Key contribution from an AI perspective

To address your concern, we clarify four points: we develop novel AI methods for this problem (a); our experiments yield scientific insights into how memory can be modeled and acted upon (b); the methods transfer across models, datasets and evaluation protocols (c); and the work belongs to an established line of research at major AI venues (d).

(a) What the work contributes [W1, Q1].

The reviewer's plain-language summary is accurate: we fit a forgetting curve (half-life) from review histories and use it to decide what to review each day under a fixed budget. "OSR" simply abbreviates Open-Space Retrieval, our name for the retrieval-based correction layer. We develop novel AI methods for this problem. From an AI perspective, the work contributes three techniques that are not tied to a particular dataset:

  1. Relative-target parameterization. When the same latent state is observed at many query times, we regress log⁡(h/t)\log(h/t) instead of hh. This turns multiplicative scale variation into an additive offset and applies to any regression problem in which one state is observed at several query times.
  2. Retrieval-augmented residual correction under an explicit information boundary. On top of any learned base model, a deterministic, non-parametric layer transfers residual evidence from related historical states. Corrections are restricted to sources that the evaluation protocol allows the model to see, and each one records its source state, relation and weight.
  3. Budgeted decision-making from predictions. OSR-ACRI converts per-item estimates into a capacity-constrained ranking, using the marginal cost of one more day of delay plus queue age.

(b) Scientific insight [W1].

Our experiments reveal three findings about how human memory can be modeled and acted upon:

  1. Memory-model errors are shared across related histories. The same residual transfer improves all 27 baselines, from formulaic to Transformer models, on all three datasets, so these errors can be corrected from related states whatever the model.
  2. Memory strength can be separated from query timing. Predicting log⁡(h/t)\log(h/t) isolates strength and helps on both MaiMemo and Anki in a matched factorial.
  3. Better estimates yield better decisions. Prediction error tracks 365-day scheduling regret across the 27 baselines (ρ = 0.98).

(c) The design is neither dataset-specific nor model-specific [W1].

The concern that the method is tailored to one dataset is directly testable, and three results show that it transfers across models, data sources and evaluation protocols:

  • Model-agnostic. Adding the retrieval layer lowers H-SMAPE for each of the 27 baselines on all three datasets, with median relative reductions of 8.3%, 11.6% and 4.9%. A rule hand-tuned to one dataset or one model would not transfer across all 81 baseline–dataset pairs.
  • Same method, heterogeneous data. The same relation family and code run on two native spaced-repetition logs (MaiMemo, Anki) and a knowledge-tracing log (EdNet-Recall). Only scalar constants are set per source, and all of them are selected on Validation.
  • Strict generalization. OSR-HLR has the lowest error among 27 baselines in all 15 dataset–protocol settings, including the Joint protocol, which removes learner, item and state overlap between training and test, and on separately held-out Anki users.

(d) Relevance to the AI community [W1, Q1].

Memory modeling and review scheduling are an established, widely cited topic at major AI venues. Half-life regression was introduced at ACL (Settles & Meeder, 2016), deep knowledge tracing at NeurIPS (Piech et al., 2015; over 1,800 citations on Semantic Scholar) and optimized spaced-repetition scheduling at KDD (Ye et al., 2022), and recent memory and forgetting models continue to appear at AAAI and KDD (Bai et al., 2025; Sun et al., 2025).

AuthorsOpenReview ↗

Response Part 2/2

W2 — Practical verification and the use of simulation

"Half of the claim is built around ACRI, which is only simulated"

(a) The core claims are verified on real learners' outcomes.

Most of the paper's evidence does not rely on simulation. OSR-HLR's predictions are evaluated on held-out real reviews from three platforms, under 15 dataset–protocol settings and on separately held-out Anki users. The decisions these predictions induce are also scored on real data: in the next-review evaluation, each interval chosen by a shared threshold scheduler is evaluated against a half-life independently fitted from real held-out reviews, and OSR-HLR reduces target-recall error by 19.7% and late-review shortfall by 23.1%. Simulation is used only for the 365-day analyses, where it is the only way to follow many predictors and schedulers on the same learners over a year.

(b) The simulation is grounded in real-world data and enables comparisons a live trial cannot.

Every component of the simulation comes from real learners: the predictors are trained on real review logs, the initial memory states are taken from real held-out learners, and the transition environments are fitted on real review transitions. Simulation then offers three advantages. First, every policy is evaluated on exactly the same memory states and random outcomes, a paired counterfactual comparison that a live trial cannot provide, because a real learner can follow only one schedule. Second, it makes large sensitivity analyses feasible, such as the four transition environments and the 105-setting scan below. Third, it exposes no real learner to an untested policy.

(c) Future work: verification in a deployed system.

To further address the reviewer's concern, we will explore real-world deployment and seek collaboration with a learning platform for a prospective study. This is feasible: related memory models already run in Duolingo (Settles & Meeder, 2016) and Anki (FSRS, one of our baselines), DHP/SSP-MMC were developed with the MaiMemo app (Ye et al., 2022; Su et al., 2023), and our methods use the same kind of review logs.

"The simulation uses the ground-truth model, hence the improvement is expected"

(a) No policy reads the simulator's ground truth.

In the scheduling comparison, every scheduler receives the same predictor, not the simulator's latent half-life. The latent half-life is hidden from all policies, which observe only forecasts, completed outcomes and queue age. All policies also share the same initial states, daily capacity, transition environment and random streams, so the simulator treats every method identically and the only difference is the allocation rule. OSR-ACRI's advantage therefore cannot come from knowing the answer.

(b) The advantage is robust across environments and predictors.

The simulator is fitted independently from Train transitions, with four environments (ridge and gradient-boosting growth models, each in a standard and a pessimistic variant). In the headline 365-day comparison over all four environments at matched workload, OSR-ACRI's paired gain over Threshold is +0.056 retention AUC (95% CI [+0.051, +0.060]; 14/16 paired decks improved). In a further 105-setting scan over three predictors unrelated to the simulator (DHP, GRU-HLR, FIFAKT), five daily capacities and seven target-recall levels, using two of these environments, OSR-ACRI ranks first among seven schedulers in 101/105 settings for retention AUC, 102/105 for time below target and 87/105 for final recall.

Reviewer_SBfBOpenReview ↗

Thanks

I thank the authors for the feedback.

I think this work is very applied, and more like a field study for data science in education / cognition. The AI and math aspect in this work is not convincing, but I am not qualified to judge this work in its particular applied field. I still recommend rejection as the topic is a bit too focused, but I would also hope the AC to seek feedback from experts in the field.

Reviewer_KGYZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines half-life prediction, retrieval-based residual correction, and capacity-aware scheduling. Half-life prediction is evaluated on three datasets. The paper also studies review decisions through fitted simulations and examines the effect of history re-encoding in a separate loop.

Critical strengths
  1. Split-specific targets, shared postprocessing, and additional held-out-user evaluation address important fairness concerns and strengthen the predictive evidence.

  2. Prediction improves across all three datasets, and the ablations indicate that both the relative target and residual correction are useful. These results provide a substantive empirical case for acceptance, especially on MaiMemo and EdNet.

Critical weaknesses
  1. Key retrieval and scheduling components remain incompletely specified. Equation 7 does not fully define the relation weights, support functions, and clipping constants. Furthermore, the online predictor's query lag and support inputs, fitted transition rules, and the reference schedulers' ranking rules under capacity constraints need clarification. These omissions make the proposed prediction-to-decision system difficult to assess or reproduce.

  2. The attribution of the predictive margin to the relative target goes beyond what the ablation results establish. For MaiMemo, removing the relative target costs 0.352 points compared with a 2.83-point baseline gap. For the five-protocol EdNet comparison, the removal costs 0.874 points compared with a 17.22-point baseline gap. Anki differs, as its removal penalty exceeds its baseline gap. The attribution in §5.2 and Appendix I therefore needs to be reconciled with the matched-target description in Appendix F. Furthermore, the positive MaiMemo interaction indicates diminishing marginal gains under the removal-minus-full definition rather than establishing complementarity. The target benefit is 0.738 points without residual correction but only 0.352 points with it.

Questions for authors
  1. [Clarification] Could the instantiated retrieval rule and one complete scheduling step be provided, including the online predictor inputs and the reference schedulers' handling of capacity constraints?

  2. [Clarification] How should the predictive-margin and complementarity claims be reconciled with the reported ablation results?

10 comments
AuthorsOpenReview ↗

Response Part 1/2

We thank the reviewer for the detailed review and for recognizing that split-specific targets, shared postprocessing and held-out-user evaluation strengthen the predictive evidence.

W1 / Q1 — Instantiated retrieval rule and one complete scheduling step

(a) Relation weights, support and clipping (instantiated Equation 7) [W1, Q1]. For a state vv without an exact visible label, relation kk uses its compatible visible sources SvS_v. Residuals are measured against the same uncorrected neural prediction, eu=log⁡(huvisible/h^u(0))e_u=\log(h_u^{\mathrm{visible}}/\hat h_u^{(0)}), and clipped before averaging:

mv=∑u∈Svρuclip⁡(eu,−C,C)∑u∈Svρu⋅∣Sv∣∣Sv∣+s,m_v=\frac{\sum_{u\in S_v}\rho_u\operatorname{clip}(e_u,-C,C)}{\sum_{u\in S_v}\rho_u}\cdot\frac{\lvert S_v\rvert}{\lvert S_v\rvert+s},

ρu=clip⁡!(cumax⁡(c~,1),.25,4)p,\rho_u=\operatorname{clip}\\!\Big(\frac{c_u}{\max(\tilde c,1)},.25,4\Big)^{p},

with source support cuc_u and median visible support c~\tilde c. A relation is supported iff Sv≠∅S_v\neq\emptyset; otherwise it contributes zero. The prediction is multiplied by exp⁡(αmv)\exp(\alpha m_v), relation by relation in the order below. Cross-difficulty also weights sources by e−γσumax⁡(∣du−dv∣,1)−ηe^{-\gamma\sigma_u}\max(\lvert d_u-d_v\rvert,1)^{-\eta} and shrinks them by (cu/(cu+τ))1/2(c_u/(c_u+\tau))^{1/2}.

Relation (in order)KeyMaiMemoAnki
Cross-difficultysame history, any difficultyα .35, C .25, s 10, p 1, η 1, τ 500, γ 16α .5, C 30, s 3, p .5, η 2, τ 0, γ 1
Suffix-4last ≤4 outcomes + interval binsα .175, C .25, s 1, p 1α .35, C .25, s 5, p 0
Ancestor-2history minus last 2 pairsα .05, C .25α .05, C .25
Suffix-8last ≤8 outcomes + interval binsα .7, C .1, s 10, p 1α .7, C .2, s 40, p 0
Difficulty slopesame history; ridge slope (λ 100)α .4, C .05α .8, C .05
Suffix-2last ≤2 outcomes + interval binsα .7, C .1, s 3, p 0α .7, C .1, s 3, p 0
Parent measurementhistory minus last pairα .0125, C .05α .05, C .05
Sign suffix-8difficulty + last ≤8 signs + binsα .05, C .2, s 3, p 0α .1, C .2, s 100, p 0

Interval bins are round⁡(blog⁡(1+t))\operatorname{round}(b\log(1+t)) with bb = 3/4 (suffix-4, MaiMemo/Anki), 4 (suffix-8), 2/.5 (suffix-2) and 1 (sign suffix-8). On MaiMemo, suffix-4 is off at depth ≤4 or 13–15 and suffix-2 at depth ≥13. EdNet-Recall uses the same rules with cross-difficulty (α, C, s, p, η, τ, γ) = (.275, .25, 3, 0, 1, 0, 0); (α, C, s, p) = (.35, .25, 5, 0) for suffix-4, (.7, .1, 20, 0) for suffix-8, (.7, .1, 3, 0) for suffix-2 and (.05, .2, 3, 0) for sign suffix-8; slope α .8; ancestor .05/.25. Test queries use only Train/Validation sources.

(b) Online predictor inputs and query lag [W1, Q1]. The scheduler queries the neural ensemble

h^=exp⁡(13∑s∈43,47,51log⁡h^s)\hat h=\exp\left(\frac{1}{3}\sum_{s\in\\{43,47,51\\}}\log\hat h_s\right)

with the completed outcome/interval history, difficulty and a candidate lag; support counts are one, and the simulator's latent half-life is never an input. From the stored forecast h[0]h^{[0]}, for j=0,1j=0,1:

b[j]=max⁡1,rint⁡(−h[j]log⁡2q),b^{[j]}=\max\\{1,\operatorname{rint}(-h^{[j]}\log_2q)\\},

h[j+1]=h[j]h^(x,b[j]).h^{[j+1]}=\sqrt{h^{[j]}\hat h(x,b^{[j]})}.

New items start at the Train median.

(c) One complete scheduling step [Q1]. On day tt:

  1. Rank due items by

Ii=(t−di)+12e−4[cold]+1−q([q−p^i(ai+1)]+−[q−p^i(ai)]+),I_i=(t-d_i)+\frac{12e^{-4[c_{\mathrm{old}}]_{+}}}{1-q}\left([q-\hat p_i(a_i+1)]_{+}-[q-\hat p_i(a_i)]_{+}\right),

with p^i(a)=2−a/hi\hat p_i(a)=2^{-a/h_i}.

  1. Review the top KK, ties by item index; the rest stay overdue.

  2. Observe yiy_i.

  3. Update

cnew=.9cold+.1mean⁡(p^i−yi).c_{\mathrm{new}}=.9c_{\mathrm{old}}+.1\operatorname{mean}(\hat p_i-y_i).

  1. Re-query as in (b) and set

d_{\mathrm{next}}=t+\max\left\\{1,\operatorname{rint}\left(e^{-4[c_{\mathrm{new}}]_{+}}\max\\{1,\operatorname{rint}(-h^{[2]}\log_2q)\\}\right)\right\\}.

In the recorded day-1 trace (K=32K=32), 119 items are due, 32 reviewed, 87 remain overdue, and the EMA moves from 0 to 0.0247; e.g., item 3 (priority 17.70) is answered incorrectly, gets forecast 1.57 days and is due on day 2.

(d) Reference schedulers under capacity [W1, Q1]. All references share earliest-due-first ranking, item-index ties and the hard daily limit; intervals are rounded and at least one day. Proposals: Threshold −h^log⁡2q-\hat h\log_2q; Half-life h^\hat h; MEMORIZE inverse-sampled wait with rate 1/h^1/\hat h, qmax⁡=2.56q_{\max}=2.56; SSP-MMC published DHP table (target 360 days); Anki max⁡(2.5−.15,lapses,1.2)streak\max(2.5-.15\\,\text{lapses},1.2)^{\text{streak}}; Random 1+⌊Umax⁡(rint⁡h^,1)⌋1+\lfloor U\max(\operatorname{rint}\hat h,1)\rfloor.

(e) Fitted transition rules [W1]. Fitted on Train transitions:

hnext=clip⁡(h,eclip⁡(F(ϕ),−2,2),.05,3650),h_{\mathrm{next}}=\operatorname{clip}\left(h\\,e^{\operatorname{clip}(F(\phi),-2,2)},.05,3650\right),

with

ϕ=(y,log⁡(1+a/h),log⁡h,d/10,ylog⁡(1+a/h)),\phi=(y,\log(1+a/h),\log h,d/10,y\log(1+a/h)),

where FF is ridge or gradient boosting, each with a pessimistic variant using Validation-selected outcome offsets.

AuthorsOpenReview ↗

Response Part 2/2

W2 / Q2 — Predictive margin and interaction sign

(a) Removal penalties and system gaps are different estimands [W2, Q2]. A removal penalty is a conditional effect inside the full model, whereas the baseline gap compares two complete implementations. Their ratio is therefore not a contribution share, whether 0.352 vs. 2.83 pp on MaiMemo or 0.874 vs. 17.22 pp on EdNet-Recall. The EdNet figure is moreover a descriptive average over five protocols with different held-out populations, so it cannot be decomposed by a single removal penalty. For the same reason, a removal penalty can exceed the system gap, as on Anki. The four-factor control shows why: on Anki the engineered features act in the opposite direction (−2.58 pp), so a single positive component can exceed the end-to-end gap.

(b) Consistency with the matched-target comparison [W2, Q2]. Once a comparator shares the same relative target, inputs, heads and residual-transfer path, the remaining encoder difference is small (adapted − OSR: +0.02 ± 0.07 pp on MaiMemo, −0.54 ± 1.05 pp on Anki; parameter-matched LTKT/FIFAKT-style encoders, 10 seeds). The complete baseline gap is thus not attributed to the target parameterization. The relative target is a conditional effect within the system, and its size is measured directly by a 2⁴ width- and input-shape-matched factorial over relative target, engineered features, local heads and residual transfer (signed Shapley H-SMAPE reduction in pp, mean ± sample SD over 10 seeds):

DatasetRelative targetResidual transferEngineered featuresLocal heads
MaiMemo+0.35 ± 0.12+0.49 ± 0.03+0.13 ± 0.11+0.10 ± 0.15
Anki+2.11 ± 0.79+3.36 ± 0.10−2.58 ± 0.50−0.49 ± 0.58

The relative target and residual transfer are the two factors that are positive on both datasets.

(c) Interaction sign [W2, Q2]. Let L11L_{11}, L01L_{01}, L10L_{10} and L00L_{00} denote the errors of the full model, target removed, residual removed and both removed. The removal-defined interaction is

I=(L00−L11)−(L01−L11)−(L10−L11).I=(L_{00}-L_{11})-(L_{01}-L_{11})-(L_{10}-L_{11}).

I>0I>0 is equivalent to L00−L10>L01−L11L_{00}-L_{10}>L_{01}-L_{11}: the target benefit is larger without residual correction (0.738 vs. 0.352 pp on MaiMemo). The four-factor control gives the same reading under the opposite sign convention,

J=mean⁡other factors[(L01−L11)−(L00−L10)],J=\operatorname{mean}_{\text{other factors}}\left[(L_{01}-L_{11})-(L_{00}-L_{10})\right],

with J = −0.38 pp on MaiMemo and −0.86 pp on Anki. Both indicate diminishing marginal gains: each mechanism adds less when the other is already present.

Dear Reviewer,

We appreciate your time and insightful comments. Since the discussion period wraps up in about a day, we kindly invite you to take a look at our rebuttal and additional experiments when you have a moment. We remain available to address any further questions ahead of the deadline.

Thank you for your consideration,

The Authors

Reviewer_ftxkOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a closed-loop framework for spaced repetition that couples half-life prediction with capacity-aware scheduling. OSR-HLR predicts log(h/Δt) rather than raw half-life, uses a masked BiGRU with outcome-aware and multiscale history features, aggregates observations by memory state, and applies exact-state lookup plus an ordered set of deterministic residual-transfer relations. The retrieval layer is restricted to protocol-visible historical states and records the source, relation, and weight of each correction. OSR-ACRI turns the resulting half-life estimates into target-recall intervals and ranks due items using one-sided online calibration, marginal delay cost, and queue age under a fixed daily review budget. The authors evaluate on MaiMemo, Anki, and EdNet-Recall against 27 baselines. Under source-only protocols, OSR-HLR reports the best H-SMAPE in all 15 dataset–protocol settings, with Row-protocol reductions of 17.40%, 2.59%, and 42.46%, respectively. With a fixed scheduler, the predictor reduces 365-day regret by 8.7%–15.8%; with the predictor and workload fixed, OSR-ACRI improves retention AUC and final recall relative to Threshold. The study is careful about label visibility and provides extensive ablations, but the scheduling evidence remains simulation-based, and the independent contribution of the proposed prediction components could be better isolated.

Critical strengths
  1. Strong information-boundary discipline and generalization design. The main source-only protocols refit half-life labels within each split and restrict retrieval to Train/Validation states; the held-out Test outcome is not used for training, retrieval, correction, or model selection. The Row/Learner/Item/State/Joint protocols and the additional held-out-Anki identity and chronological evaluations make the prediction evidence more credible than a single random split. The manuscript also clearly separates its transductive diagnostic views from the headline source-only claims.
  2. Broad, decision-oriented evaluation with useful component isolation. The paper compares 27 baselines across three data sources and evaluates both prediction error and downstream scheduling. Holding the scheduler fixed when evaluating predictors, and holding the predictor/workload fixed when comparing schedulers, is a sensible design for separating the two effects. The model-agnostic retrieval analysis and the core ablations provide useful evidence beyond a single headline metric.
Critical weaknesses
  1. Novelty and attribution of the prediction gains are not fully resolved. OSR-HLR combines a relative-target parameterization, a feature-rich BiGRU encoder with hand-designed outcome-aware/multiscale statistics, and a fixed ordered residual-transfer layer. The auditable deterministic transfer is practically appealing, but the paper should more clearly distinguish conceptual novelty from careful feature engineering and a hand-specified correction pipeline. The current core ablations show that the relative target and residual graph help, but they do not fully explain the full gap to the strongest baselines, particularly when the encoder uses a substantial custom feature set. A feature-matched or architecture-matched factorial ablation would strengthen the attribution. In addition, the Row-protocol H-SMAPE gain on Anki is only 2.59%, and the paper itself notes weaker support for Anki Row/Item/Joint settings.
  2. The scheduler claim is still dependent on simulated transitions, and calibration appears to contribute little. The 365-day scheduling results use source-fitted simulated environments rather than prospective outcomes from real learners, so they do not establish causal learning benefits or robustness to transition-model misspecification. The authors acknowledge this limitation, which is appropriate, but it narrows the practical claim. Moreover, the scheduler ablation reports nearly unchanged AUC and time-below-target when calibration is removed, and slightly higher final recall without calibration; queue age appears to be the largest contributor. The paper should more explicitly qualify the role of calibration and provide paired uncertainty/sensitivity results for the headline scheduler gains.
Questions for authors
  1. Can you provide a feature- and architecture-matched factorial ablation that separates the relative-target parameterization, the hand-designed outcome-aware/static features, the local correction heads, and the residual-transfer layer? Which component accounts for the remaining advantage over the strongest baseline, especially on MaiMemo where removing either core mechanism alone changes H-SMAPE by only a few tenths of a point?
  2. For OSR-ACRI, can you report paired confidence intervals for the differences against Threshold and MEMORIZE, and sensitivity to alternative transition environments, daily capacities, and target-recall levels? Given that removing calibration barely changes AUC/time-below-target and slightly increases final recall, what specific measurable benefit does calibration provide?
  3. How much of the raw review data and how many distinct learners/items are retained after the minimum-support and label-construction filters? Do the prediction and scheduling conclusions persist under alternative support thresholds or learner-weighted evaluation, given that the main estimand weights aggregated observations equally?
4 comments
AuthorsOpenReview ↗

Response (Part 1/3)

We sincerely thank the reviewer for the careful and constructive review, and for recognizing our information-boundary discipline and decision-oriented evaluation. The requested factorial attribution, paired scheduler uncertainty and learner-weighted analyses have strengthened the paper, and we address each point below.

R1-W1 / R1-Q1 — Novelty and attribution of the prediction gains

(a) What is conceptual and what is engineered.

The conceptual contribution consists of two mechanisms:

  1. Relative-state estimation. The model predicts log(h/t) instead of h, so that state strength is separated from query timing.
  2. Protocol-visible residual transfer. Corrections come only from Train/Validation states, and every correction records its source state, relation and weight.

The BiGRU encoder, the hand-designed outcome/multiscale statistics, the local correction heads and the specific relation constants are one instantiation of this interface. We do not claim them as contributions.

(b) The factorial control supports this split.

Setup. We ran a 2⁴ factorial on MaiMemo and Anki (source-only Row protocol, seeds 42–51) crossing relative vs. direct target, engineered vs. minimal inputs, local heads on vs. off, and full vs. exact-only residual transfer. All 16 cells keep hidden width 128, the same 11-channel event tensor and the same 100-column static branch. Minimal inputs mask the engineered channels rather than deleting them, so the comparison is width- and input-shape-matched. Data, split, optimizer and the Validation selection rule are shared, and Test labels are never used for training, retrieval or selection.

Result. Signed Shapley decomposition of the all-off → full change (H-SMAPE reduction in pp, mean ± sample SD over 10 seeds; positive = helps):

DatasetRelative targetResidual transferEngineered featuresLocal heads
MaiMemo+0.35 ± 0.12+0.49 ± 0.03+0.13 ± 0.11+0.10 ± 0.15
Anki+2.11 ± 0.79+3.36 ± 0.10−2.58 ± 0.50−0.49 ± 0.58

The two conceptual mechanisms are positive on both datasets, and residual transfer is the most stable factor. The engineered features and local heads are small on MaiMemo and adverse on Anki.

Encoder control. We replaced only the encoder with parameter-matched LTKT/FIFAKT-style encoders, keeping inputs, target, heads, loss, optimizer, selection and the residual-transfer path fixed:

DatasetAdapted encoder (params vs. OSR)Adapted − OSR H-SMAPE (pp)OSR better in
MaiMemoLTKT (+1.05%)+0.02 ± 0.076/10 seeds
AnkiFIFAKT (−0.52%)−0.54 ± 1.054/10 seeds

The encoder change is a near tie on MaiMemo and favors the adapted FIFAKT encoder on Anki. We therefore do not attribute the gain to the BiGRU or to the feature set.

Which component accounts for the remaining advantage? None alone. On MaiMemo, the full gap to the strongest baseline is 2.83 ± 0.50 pp. The four factors account for 1.08 pp, and the remaining baseline → all-off term is 1.75 ± 0.56 pp. That term is not an encoder effect: the strongest baseline already uses the full shared residual postprocessor, while the all-off cell uses exact-only lookup. It therefore mixes postprocessing endpoint, scaffold and training-interface differences, and we do not assign it to any single module. The interaction between relative target and residual transfer is negative (J = −0.38 pp on MaiMemo, −0.86 pp on Anki), so each mechanism has a smaller marginal gain when the other is present. This indicates diminishing marginal gains, not causal synergy.

(c) Anki.

Anki is the most demanding of the three sources: its items are learner-specific cards (learner_id:card_id), so no item identity is shared across learners, and the strongest baselines are tightly clustered (FIFAKT 39.34, DGMKT 39.38, SPsyINN 39.74 H-SMAPE). In this setting OSR-HLR (38.32) has the lowest mean H-SMAPE in all five source-only protocols (1.54–6.40% reductions), and its Row margin over the strongest baseline (1.02 pp) is larger than the spread among the top three baselines (0.40 pp).

The factorial also shows where the Anki margin comes from. The two core mechanisms contribute their largest effects on Anki (residual transfer +3.36 pp, relative target +2.11 pp, +5.48 pp combined), while the engineered features act in the opposite direction (−2.58 pp). The core mechanisms therefore transfer to Anki, and the feature instantiation, which we do not claim as a contribution, is what narrows the end-to-end margin.

Two separately held-out Anki populations give an independent check: OSR-HLR is 11.36% (identity isolation) and 2.40% (within-learner chronological) below the strongest per-protocol baseline among the 27 baselines.

AuthorsOpenReview ↗

Response (Part 2/3)

R1-W2 / R1-Q2 — Paired scheduler uncertainty, sensitivity and the role of calibration

(a) What the scheduler comparison isolates. The scheduling experiments are designed to isolate allocation quality: all policies share the same predictor, initial states, daily capacity, transition environment and random streams, so differences come only from which due items each policy reviews. The policies observe only forecasts, completed outcomes and queue age; none of them reads the simulator's latent half-life or its update rule.

(b) Paired uncertainty for the headline gains and alternative transition environments. For the headline 365-day comparison (512 states, K = 32, q = 0.90, workload matched at 11,680 reviews for every policy), we resampled whole paired decks across four transition environments × four rollouts (16 paired decks; 5,000 bootstrap draws, stratified by environment). All gains favor OSR-ACRI:

ReferenceMetricMean paired gainPaired 95% CIDecks improved
ThresholdRetention AUC+0.0561[+0.0512, +0.0602]14/16
ThresholdTime-below reduction+0.0526[+0.0480, +0.0569]14/16
ThresholdFinal recall+0.0937[+0.0822, +0.1049]14/16
MEMORIZERetention AUC+0.0772[+0.0708, +0.0831]16/16
MEMORIZETime-below reduction+0.0432[+0.0379, +0.0487]11/16
MEMORIZEFinal recall+0.1308[+0.1187, +0.1429]15/16

The four environments use two different growth models (ridge and gradient boosting), each in a standard and a pessimistic variant. The per-environment mean gain is positive in 10 of 12 environment × metric cells against Threshold and in 10 of 12 against MEMORIZE.

(c) Sensitivity to capacity, target recall and predictor. We further evaluated 3 predictors (DHP, GRU-HLR, FIFAKT) × K ∈ {24, 32, 48, 64, 96} × q ∈ {0.825, 0.850, 0.875, 0.900, 0.925, 0.950, 0.975}, giving 105 settings (64 states, 2 environments × 3 rollouts, 100 days). Within each setting, all seven schedulers share the predictor, capacity, initial states and random streams. OSR-ACRI ranks first among the seven schedulers in 101/105 settings for retention AUC, 102/105 for time below target and 87/105 for final recall. Grid-average paired gains (setting-level bootstrap, 10,000 draws) are:

ReferenceRetention AUCTime-below reductionFinal recall
Threshold+0.0566 [+0.0486, +0.0650]+0.0994 [+0.0862, +0.1127]+0.0278 [+0.0223, +0.0340]
MEMORIZE+0.1203 [+0.1142, +0.1264]+0.1875 [+0.1729, +0.2027]+0.0651 [+0.0576, +0.0730]

The direction therefore holds across predictors of different quality, daily capacities and target-recall levels, in addition to the four transition environments in (b).

(d) The specific role of calibration. Calibration plays a different role from the two ranking terms. The marginal delay index and queue age determine which due items are served (removing the delay index reduces OSR-ACRI to Threshold-level performance, and removing queue age lowers AUC by 0.174). Calibration instead protects against overconfident forecasts: after outcomes fall below prediction, it shortens the next intervals so that items do not drift below the target. Its measurable effect is therefore on time below target:

SettingMetricFull − No calibrationPaired 95% CI
Headline, matched workloadTime-below reduction+0.0122[+0.0087, +0.0156]
Headline, matched workloadRetention AUC+0.0038[+0.0004, +0.0074]
Headline, matched workloadFinal recall−0.0019[−0.0063, +0.0022]
105-setting gridTime-below reduction+0.1011[+0.0877, +0.1149]

At identical workload, calibration lowers time below target with an interval that excludes zero, while leaving final recall unchanged. The reduction is largest in the pessimistic gradient-boosting environment (+0.044), where outcomes are systematically worse than the predictor expects, which is exactly the overconfidence case the one-sided update targets. In the 105-setting grid, calibration reduces time below target in 104/105 settings, and its effect is largest at lower target recall (+0.176 at q = 0.825), where each interval is longest and an overconfident forecast is most costly.

AuthorsOpenReview ↗

Response (Part 3/3)

R1-Q3 — Retained data, support thresholds and learner weighting

(a) Retained data. Counts are computed from the raw event membership of the final rows, after label construction, support filtering and row caps.

DatasetRaw eventsEvents represented by final Row cohortFinal learnersFinal items
MaiMemo226,363,47130,791,756701,98517,080
Anki67,611,0153,943,7971,0003,154,076
EdNet-Recall95,293,92610,271,125155,88312,209

MaiMemo and EdNet items are global word/question IDs; Anki items are learner-specific cards (learner_id:card_id), which is why the Anki item count is large. The retained cohorts cover hundreds of thousands of learners on MaiMemo and EdNet and the full pinned 1,000-learner Anki cohort.

(b) Alternative support thresholds. Changing the threshold changes which reviews define each state, so filtering existing rows would not reproduce its effect. For each dataset we therefore rebuilt the split-local sufficient statistics and half-life labels from raw events at two additional thresholds (halved and doubled), and cold-start retrained both OSR-HLR and the strongest original baseline with ten paired seeds (120 new runs; the default thresholds reuse the verified original runs). OSR-HLR has the lower equal-row H-SMAPE at all 9 dataset × threshold combinations:

DatasetSupport (cell/state)BaselineOSR-HLR
MaiMemo50/50 · 100/100 · 200/20020.50 · 16.26 · 13.4018.44 · 13.43 · 9.68
Anki2/50 · 5/100 · 10/20046.68 · 39.34 · 34.1845.34 · 38.32 · 32.99
EdNet-Recall10/50 · 20/100 · 40/20049.91 · 59.26 · 78.0332.03 · 34.10 · 41.35

(c) Learner-weighted evaluation. We average each learner's represented losses before giving every retained learner equal weight,

Lmacro=1∣U∣∑u∈U∑jnujℓj∑jnuj,L_{\mathrm{macro}}=\frac{1}{\lvert U\rvert}\sum_{u\in U}\frac{\sum_j n_{uj}\ell_j}{\sum_j n_{uj}},

where nujn_{uj} is learner uu's raw contribution to row jj. Because half-life labels are state-level estimates pooled across many learners' reviews, the learner-level question is answered most directly by each learner's observed outcomes. On these observed losses, OSR-HLR has the lower learner-macro Brier score in 9/9 and the lower learner-macro NLL in 8/9 dataset × threshold combinations. The complete learner-macro results (mean ± sample SD over seeds; baseline / OSR-HLR) are:

DatasetSupportH-SMAPE (%)Observed NLLObserved Brier
MaiMemo50/5016.58 ± 1.10 / 6.91 ± 0.510.3980 ± 0.0018 / 0.3945 ± 0.00040.1233 ± 0.0009 / 0.1222 ± 0.0002
MaiMemo100/10015.88 ± 1.13 / 4.49 ± 0.510.4434 ± 0.0015 / 0.4394 ± 0.00030.1418 ± 0.0006 / 0.1407 ± 0.0001
MaiMemo200/20025.03 ± 0.81 / 5.11 ± 0.530.4532 ± 0.0019 / 0.4389 ± 0.00040.1458 ± 0.0008 / 0.1401 ± 0.0002
Anki2/5020.41 ± 2.85 / 22.77 ± 4.920.5956 ± 0.0047 / 0.5913 ± 0.00830.1153 ± 0.0003 / 0.1145 ± 0.0002
Anki5/10023.30 ± 3.53 / 26.91 ± 11.690.5695 ± 0.0082 / 0.5485 ± 0.01520.1158 ± 0.0005 / 0.1144 ± 0.0006
Anki10/20022.54 ± 2.57 / 23.51 ± 2.210.5379 ± 0.0038 / 0.5304 ± 0.00420.1158 ± 0.0002 / 0.1150 ± 0.0003
EdNet-Recall10/5033.94 ± 2.10 / 6.30 ± 0.801.6465 ± 0.0343 / 1.6282 ± 0.00600.3440 ± 0.0011 / 0.3421 ± 0.0002
EdNet-Recall20/10034.77 ± 0.90 / 6.34 ± 0.511.6146 ± 0.0115 / 1.6174 ± 0.00920.3432 ± 0.0004 / 0.3416 ± 0.0004
EdNet-Recall40/20035.14 ± 1.30 / 9.67 ± 5.131.6237 ± 0.0189 / 1.6066 ± 0.01620.3439 ± 0.0006 / 0.3414 ± 0.0006

The EdNet-Recall 20/100 cohort uses three seeds; all other cohorts use ten.

(d) Scheduling. We rebuilt the MaiMemo scheduling population at three support levels (50/100/200) and replayed the fixed predictor and simulator with both equal-state and learner-weighted aggregation (16 paired decks each). Against Threshold and MEMORIZE, OSR-ACRI has positive mean gains in all 36 support × weighting × reference × metric contrasts.

AuthorsOpenReview ↗

Summary of Responses to Reviewer ftxk

Dear Reviewer ftxk,

With less than an hour remaining before the discussion deadline, we have summarized our responses below for your convenience.

  1. Attribution of prediction gains (Q1). The matched four-factor ablation showed that the relative target and protocol-visible residual transfer helped on both MaiMemo and Anki. The four factors accounted for 1.08 percentage points of MaiMemo’s 2.83-point baseline gap. The remainder reflects several differences between the complete systems.

  2. Scheduler uncertainty and sensitivity (Q2). We reported paired bootstrap intervals against Threshold and MEMORIZE across four simulated transition environments. Across 105 settings varying the predictor, daily capacity, and target recall, OSR-ACRI ranked first in 101 for retention AUC, 102 for time below target, and 87 for final recall.

  3. Role of calibration (Q2). Calibration shortened review intervals after overconfident forecasts and reduced time below target in 104 of 105 settings. Final recall was comparable at the same workload in the headline comparison.

  4. Data retention and support thresholds (Q3). We reported retained review, learner, and item counts and rebuilt labels at half and twice the default support threshold. OSR-HLR achieved lower H-SMAPE in all nine dataset–threshold combinations.

  5. Learner-weighted evaluation (Q3). OSR-HLR had lower learner-macro Brier scores in 9 of 9 comparisons and lower NLL in 8 of 9. On MaiMemo, OSR-ACRI showed positive mean gains in all 36 scheduling comparisons across support thresholds and weighting schemes.

We hope our responses and additional analyses will be helpful in the next stage of review. Thank you again for your thoughtful questions and careful consideration of our work.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.