acceptodds
Under review as a conference paper at ICLR 2027

AIRC: Action-Innovation Response Certificates for Risk-Aware Offline Policy Improvement in Blood Glucose Control

Abstract

Offline insulin-control learning must infer which actions are supported by retrospective treatment data, which direction improves control, and when an increase becomes unsafe. We introduce Action-Innovation Response Certificates (AIRC), an evidence architecture that converts state-relative action innovations into signed response certificates. Agreement across complementary data folds and clinically relevant prediction horizons determines the direction and confidence of policy updates, while a separate short-horizon pathway estimates the risk attached to positive insulin shifts. In SimGlucose, AIRC achieves the highest mean cumulative reward among six offline reinforcement learning baselines and improves time in range by 4.86% relative to the strongest reward baseline, IQL, while reducing the glycemia risk index by 10.77%. At the 240-minute analysis horizon, 54.67% of eligible AIRC events meet the postprandial target-range criterion, 10.2 percentage points more than with IQL. Component ablations separate the roles of response direction, asymmetric risk weighting, and innovation scaling. On matched held-out states from the public Loop dataset, AIRC improves trend–action correlation by 0.139 and directional consistency by 4.51 percentage points over IQL. Together, these results establish AIRC as a response-certified framework for risk-aware offline policy improvement in basal-insulin control. The source code and datasets are submitted as Supplementary Materials for Reproducibility.

Primary area:
reinforcement learning
Keywords:
offline reinforcement learning, risk-aware policy improvement, response certification, continuous control, automated insulin delivery

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wuC6OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes AIRC for offline basal-insulin policy learning. It uses trajectories generated by a reference Loop controller, estimates deviations in logged actions and future glucose risk, and uses their relationship to guide the learned policy. The method is evaluated mainly in SimGlucose, with an additional held-out analysis on real Loop observational data.

Critical strengths
  1. The problem is important and clinically relevant, especially because offline learning avoids unsafe online exploration.
  2. The method separates adjustment direction from low-glucose risk, which is intuitive.
  3. The experiments include several offline RL baselines and component ablations, and AIRC performs well in SimGlucose.
Critical weaknesses
  1. The residual-based response design is not sufficiently justified. In Sections 4.2–4.3, the method does not directly model how insulin affects future glucose. Instead, it relates an action residual to a future-risk residual and uses this relation to determine the policy-improvement direction. I am not convinced that the future-risk residual is a reliable signal for how insulin should be adjusted, since it can also reflect patient dynamics, meal effects, or prediction error. A direct comparison with action-conditioned future glucose prediction or trajectory prediction would help justify this design.
  2. The high- and low-glucose risk signals contain several hand-designed choices. The specific forms and parameters in Section 4.2, such as the high-glucose softplus shaping, the constants 30 and 10, the cap of 10, the low-glucose sigmoid centered at 90, and the selected 60/120 and 240/360 minute horizons, are not well justified. Since these choices affect the later value estimates, response coefficients, and actor updates, a sensitivity analysis is needed.
  3. The original Loop controller is missing from the main comparison. The SimGlucose training trajectories are generated by the reference Loop controller, and AIRC learns a policy that is anchored to those logged actions. However, Table 1 compares AIRC only with offline RL baselines and does not report the closed-loop performance of the original Loop controller. Therefore, the current experiments do not show whether AIRC actually improves over the behavior controller that generated the data. A direct Loop-vs-AIRC comparison under the same evaluation setting seems necessary.
Questions for authors
  1. Why is the residual relation in Sections 4.2–4.3 a better choice than directly modeling future glucose under different actions?
  2. How sensitive are the results to the hand-designed risk-shaping parameters and prediction horizons?
  3. How does AIRC compare directly with the original reference Loop controller in closed-loop SimGlucose evaluation?
Reviewer_32KcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes AIRC for risk-aware offline insulin policy improvement. It uses agreement across data folds and prediction horizons to guide policy updates and includes a separate pathway for estimating short-term risks.

Critical strengths

The paper addresses an important practical problem. Separating policy improvement from short-term safety assessment is a reasonable design choice. The SimGlucose results are encouraging.

Critical weaknesses

The meaning of the proposed response certificates is not sufficiently clear. Agreement across data folds and prediction horizons measures prediction stability, but does not necessarily guarantee the correctness of the action-response direction or safe policy improvement.

The action-innovation approach may still be affected by confounding in offline treatment data. The paper does not directly evaluate whether the estimated response directions agree with true causal action effects.

The real-world Loop evaluation focuses on policy behavior rather than patient outcomes. Better trend–action consistency does not necessarily establish improved glucose control or safety in real-world deployment.

Questions for authors

What formal statistical or safety guarantees do the response certificates provide?

Can the authors evaluate certificate sign accuracy and confidence calibration in simulated environments where the true action-response effects are known?

How sensitive is the certificate to the choice of data folds, prediction horizons, and unobserved confounding?

Reviewer_NYxSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes AIRC, a risk-aware offline policy learning framework for basal-insulin control using historical treatment data. The method estimates the relationship between state-relative insulin adjustments (action innovations) and future hyperglycemic burden. It then uses agreement across data folds and prediction horizons to determine the direction and confidence of policy updates, while a separate short-horizon risk component penalizes insulin increases that may lead to hypoglycemia. The method is evaluated in SimGlucose against six offline RL baselines, with additional experiments on the public Loop dataset to assess policy responses to changes in glucose trends.

Critical strengths
  • The application is important, and the design reflects real clinical asymmetry. The paper addresses risk-aware offline policy improvement for insulin dosing, where both excessive and insufficient treatment can be harmful. Separating adjustment direction, confidence, and low-glucose risk is a meaningful approach.
  • Strong empirical performance. AIRC outperforms six offline RL baselines in SimGlucose. The authors also provide paired confidence intervals for the cumulative reward improvement.
  • The ablations are informative. Removing direction raises time above range, while removing the risk penalty raises time below range and terminations, so each component has a visible and interpretable effect (Table 2). The evaluation protocol is also prespecified, with matched data, update budgets and model selection across methods.
Critical weaknesses
  • The reward is a surrogate and disagrees with the clinical metrics: in the adolescent subgroup AIRC has the highest reward but does not beat all baselines on time in range (TIR) or glycemia risk index (GRI). TIR, hypoglycemia rates and GRI should be the primary results.
  • Agreement across two folds that share the encoder and innovation model, and across two correlated horizons, does not show that the estimated direction is correct. There is neither theory nor an empirical calibration of certificate confidence.
  • Without the penalty, the certified direction gives 9.75% time below range and 13.33% terminations (20.64% and 30% in children; Tables 2 and 14), so the certificate itself pushes toward aggressive insulin increases. The method is tied to a one-dimensional basal action, glucose-specific thresholds and one simulator; the claimed generality to ordered treatment decisions is never tested.
Questions for authors
  1. Under what assumptions does ρτ\rho_\tau recover the causal effect of a basal change, given that the logged actions come from a feedback controller? Does its sign match the true dose–response in SimGlucose?
  2. How do the reference Loop controller and a behavior-cloning policy perform on the Table 1 metrics?
  3. Do the baselines improve with the same shaping signals (cHc^H, cLc^L, cSc^S)? How were the fixed hyperparameters in Table 5 chosen?
  4. What do off-policy value or predicted-hypoglycemia estimates show on held-out Loop data, and how do the logged Loop actions score on the Table 3 metrics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.