- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper applies the ACI online conformal prediction idea to the online CRO constrained problem. Instead of updating prediction coverage, this paper uses robustness-violation updating control. Since the violation is not monotone in the multi-dimensional parameter , the paper adds an external scalar to the certificate: carries the robustness control through an ACI-style update, while is updated separately to shrink the certificate. The paper provides some theorems and runs experiments on synthetic and real data.
Critical strengths
The introduction of the external scalar q to decouple the terms is interesting. Directly applying ACI-style updating has a non-nested issue, but this design brings the monotone pattern back.
Critical weaknesses
-
The paper is difficult to follow.
- At the very beginning, the paper presents an abstract problem without a friendly, example-based articulation to help readers understand the application or scenario. Without this, readers cannot imagine the potential pipeline behind this and do not know how to logically and correctly connect the prediction set, the decision, the certificate, the coverage, and the violation rate.
- The notation definition is confusing. For example, when the paper first introduces and in lines 188 and 195, it does not immediately explain their role, leaving readers unsure what they represent. Additionally, the authors could clearly define the Lipschitz and smoothness conditions in Assumption 2.
-
The experiments are too simple to fully support the claims.
- The experimental comparison (better than coverage-based ACI and PID) is not that informative or apple-to-apple, because the proposed method changes the controlled event. Specifically, in the synthetic dataset, the baselines adapt a scalar radius around a fixed center and covariance shape, but the proposed method learns them, which conflates the effects of introducing .
- Algorithm 1 explicitly mentions updating , but the real dataset experiment fixes the MLP.
Questions for authors
-
In lines 13-14, what do you mean by robustness in conformal prediction? I remember that regular conformal prediction doesn't mention robustness, except the efficiency (prediction set size) and the validity (coverage). Without clear articulation, this may confuse readers about robustness in regular conformal prediction versus OCRC.
-
Lines 428 and 429: Why for pretraining? Section 5.2 defines , and Algorithm 1 updates all of , so the MLP weights should be learned online. Otherwise, the algorithms, the theorem's ground, and the real implementation will not match.
-
For the experiments, which certificate do we report ( or )? Can we report both?
4 comments
Questions
Q1. In lines 13-14, what do you mean by robustness in conformal prediction? I remember that regular conformal prediction doesn't mention robustness, except the efficiency (prediction set size) and the validity (coverage). Without clear articulation, this may confuse readers about robustness in regular conformal prediction versus OCRC.
Response. Thank you for asking us to clarify this distinction. Here, robustness refers to the downstream decision: a decision is robust on a given round if its realized loss does not exceed the reported risk certificate. Prediction-set coverage is sufficient for this property, but outcomes outside the prediction set can also produce losses below the certificate. We now state this distinction explicitly in the abstract (lines 11–17):
We study online conditional robust optimization, where each decision minimizes the worst-case loss over a prediction set before the outcome is observed. The resulting worst-case loss serves as a risk certificate, and a decision is robust on a given round if its realized loss is at most that certificate. Online conformal methods can ensure long-run decision robustness through prediction-set coverage, but this approach can be conservative because outcomes outside the prediction set may also yield losses below the certificate.
Q2. Lines 428 and 429: Why for pretraining? Section 5.2 defines , and Algorithm 1 updates all of , so the MLP weights should be learned online. Otherwise, the algorithms, the theorem's ground, and the real implementation will not match.
Response. Thank you for highlighting this ambiguity. Pretraining provides a common initialization to reduce early instability, and the MLP weights continue to be updated online. Load-Climate-Open uses the first 40% of observations for standardization and pretraining, as clarified in Section 5.3 (lines 484–485). In the DEPI notebook, the first 40% are used to fit the MLP, the next 10% for validation, and the final 50% for online evaluation.
During online evaluation, after observing each outcome, OCRC takes a gradient step on its downstream objective to update , including the MLP weights and covariance parameters used for the next decision. This is the update specified in Algorithm 1 (lines 223–231).
Q3. For the experiments, which certificate do we report ( or )? Can we report both?
Response. Thank you for this suggestion. Each method reports the threshold it deploys: OCRC reports , whereas ACI and Conformal PID report the CRO certificate . We have clarified this convention in the captions of Figure 3 (lines 480–482) and Figure 4 (lines 514–515).
For a direct comparison within the same OCRC run, the fixed-center experiment gives a mean CRO certificate of 4.38 and a mean deployed certificate of 5.02, computed from the logged certificates and offsets. The figures report because this is the threshold deployed by OCRC.
Thanks for the authors’ revision.
- I have a concern about the modeling updates. Actually, I checked the code submission: the code
mean()returns the global networknetwork.forward(x), while the optimizer updates a deep copyself.ellipsoid.network, so OCRC's MLP does not train. This is contrary to your claim that “All methods update their MLPs”. - I checked the revised PDF but didn't find clarification for and Assumption 2: Assumption 2 (lines 276–277) is identical to the original, and lines 214–221 do not explain what weighs.
- You reported both values for B.1 (4.38 vs 5.02). Could you also report OCRC's mean alongside its mean for the main experiments? This would show the size of the added offset in each experiment.
Thank you for checking the revised manuscript and supplementary code. We address your three follow-up points below.
Q1. Online MLP updates
I have a concern about the modeling updates. Actually, I checked the code submission: the code
mean()returns the global networknetwork.forward(x), while the optimizer updates a deep copyself.ellipsoid.network, so OCRC's MLP does not train. This is contrary to your claim that “All methods update their MLPs”.
Response. Thank you for identifying this bug in the previously submitted code. As you noted, mean() called the global pretrained MLP, while the optimizer held the parameters of a separate copy. As a result, the MLP used by OCRC did not update online in that version. The figures and results in the revised manuscript were generated with the corrected implementation, in which mean() calls the same MLP instance that the optimizer updates. We have updated the supplementary code for inspection. We also reran both real-data main experiments and verified that the MLP weights of OCRC, ACI, and Conformal PID each change during online evaluation.
Q2. The role of and the definitions in Assumption 2
I checked the revised PDF but didn't find clarification for and Assumption 2: Assumption 2 (lines 276–277) is identical to the original, and lines 214–221 do not explain what weighs.
Response. You are right that the revised PDF still omits both explanations.
(a) The role of . Immediately after Eq. (3), we will explain that weights the integrated smoothed constraint residual relative to the base certificate in the primal objective. It therefore scales the contribution of robustness feedback to the update of . Appendix B.3 illustrates this role empirically. Larger values of yield smaller offsets after the trajectory stabilizes. This trend is consistent with Theorem 1, which shows that the optimal offset of the penalized problem tends to zero as .
(b) Assumption 2. We will add the following definitions immediately after the assumption. A map is -Lipschitz if for all in its domain. A differentiable map is -smooth if for all in its domain. Here is the Jacobian for the vector-valued decision map, and the derivative norm reduces to the Euclidean norm of the gradient for scalar-valued maps. These conditions apply to uniformly over , and to and uniformly over , with the constants stated in Assumption 2.
Q3. Mean base and deployed certificates in the main experiments
You reported both values for B.1 (4.38 vs 5.02). Could you also report OCRC's mean alongside its mean for the main experiments? This would show the size of the added offset in each experiment.
Response. The table reports both certificates from the same OCRC run for each main experiment, averaged over all online evaluation rounds. The last column gives the mean offset actually used when deploying the certificate.
| Main experiment | Mean | Mean | Mean |
|---|---|---|---|
| Stationary simulation | 4.2748 | 5.1894 | 0.9146 |
| Temporally dependent simulation | 4.6129 | 5.4614 | 0.8485 |
| Load-Climate-Open | −0.1375 | −0.0937 | 0.0438 |
| DEPI | 5.4312 | 5.6749 | 0.2437 |
On Load-Climate-Open, for example, the offset raises the mean certificate by 0.0438 in the experiment's loss units, from −0.1375 to −0.0937. In each row, the two certificates evaluate the same sequence of decisions.
Response to Reviewer CZYi
Thank you for your helpful comments on the exposition and experimental protocol. We have clarified the problem setup, notation, and online learning procedure, and added a comparison using the same prediction-set family. Line numbers refer to the revised manuscript.
Weaknesses
W1(a). At the very beginning, the paper presents an abstract problem without a friendly, example-based articulation to help readers understand the application or scenario. Without this, readers cannot imagine the potential pipeline behind this and do not know how to logically and correctly connect the prediction set, the decision, the certificate, the coverage, and the violation rate.
Response. Thank you for highlighting the need for a concrete example. The revised Introduction explains the full pipeline through factory electricity scheduling (lines 51–57). A prediction set represents plausible hourly load profiles. The factory selects a schedule that minimizes the worst-case cost over this set and reports that cost as its certificate. Coverage asks whether the entire realized load profile lies in the prediction set; robustness asks whether the realized cost of the selected schedule is at most its certificate. For example, a load spike during an unused hour may cause a coverage failure without changing the schedule’s cost. Thus, a failure of coverage need not imply a robustness violation.
W1(b). The notation definition is confusing. For example, when the paper first introduces Hσ and κ in lines 188 and 195, it does not immediately explain their role, leaving readers unsure what they represent.
Response. Thank you for pointing out this ambiguity. Section 3.2 now explains both quantities when they are introduced. We define the margin v as the realized loss minus the current certificate and introduce Hσ(v)=H(v/σ) as a smooth approximation to the violation indicator 1v>0. The scale σ controls the width of the transition around zero, and smoothing provides a gradient through which robustness feedback can guide updates of θ (lines 207–213). The coefficient κ determines the weight of the integrated violation penalty in the downstream objective (lines 214–221).
W1(c). Additionally, the authors could clearly define the Lipschitz and smoothness conditions in Assumption 2.
Response. Thank you for this suggestion. We have clarified the variables to which these conditions apply in Assumption 2 (lines 276–277). For ϕ(y,z), Lipschitz continuity and smoothness are with respect to z, with y held fixed. For zθ(x) and rθ(x), they are with respect to θ, with x held fixed. Smoothness means that the scalar maps have Lipschitz gradients and that the decision map has a Lipschitz Jacobian.
W2(a). The experimental comparison (better than coverage-based ACI and PID) is not that informative or apple-to-apple, because the proposed method changes the controlled event. Specifically, in the synthetic dataset, the baselines adapt a scalar radius around a fixed center and covariance shape, but the proposed method learns them, which conflates the effects of introducing qt.
Response. Thank you for identifying this potential confounding factor. We added a comparison in which all methods use the same oracle center and fixed shape and adapt only the prediction-set scale (Appendix B.1, lines 936–966, Figure 5). The robustness target is 90%.
With prediction-set flexibility held fixed, OCRC achieves robustness closest to the 90% target and a smaller mean certificate: 5.02, compared with 5.76 for ACI and 5.71 for Conformal PID. Its lower coverage is consistent with directly controlling downstream violations. These results show that the improvement persists when all methods use the same prediction-set family.
W2(b). Algorithm 1 explicitly mentions updating θ, but the real dataset experiment fixes the MLP.
Response. Thank you for pointing out the need to clarify the online updates. All methods update their MLPs during online evaluation. OCRC updates its parameters through the downstream objective, while ACI and Conformal PID take squared-error gradient steps for their mean predictors and separately update their coverage thresholds. We have clarified these updates in the revised experimental description (lines 391–395, Figures 3–4).