- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the cost of training predictors through downstream optimization problems. It focuses on strongly convex QPs with fixed Hessians and constraints, where predictions enter only the linear objective term. Exploiting their piecewise-affine solution maps, FastDFL stores local solution formulas and Jacobians and reuses them when predictions satisfy region-validity checks. Experiments demonstrate reduced computation across three tasks and two predictor architectures.
Critical strengths
-
The paper is well written and structured, with relevant background, clear motivation, and an understandable presentation.
-
The experiments demonstrate consistent acceleration. The matched comparison and construction/retrieval ablations are useful.
-
Bringing regional QP/MPC reuse into differentiable learning is a useful direction for taking advantage of recurring optimization structure.
Critical weaknesses
I generally like this work. My concerns mainly require clarification, and I would consider increasing my score if they are addressed.
-
Novelty relative to regional QP/MPC reuse. The contribution appears to be focused on DFL-specific integration and cache-management policies, rather than the underlying reuse mechanism or constant-Jacobian property. Comparisons against methods that solve every QP establish the benefit of reuse more clearly than the improvement over existing MPC regional-reuse approaches. Please clarify the differences. Can an MPC baseline be added to the experiments?
-
Restricted parameterization. How limiting are the fixed Hessian and feasible-set assumptions for intended applications? Please distinguish extensions preserving exact affine reuse from those requiring different approximations or validity checks.
-
Scalability. Appendix E.3 reports substantial memory costs, but tests only up to 192 decision variables with prediction dimension fixed at 24. Please discuss larger QPs, higher-dimensional predictions, and larger datasets, including per-sample storage costs.
-
Solver choice. The speedups are useful but mostly moderate outside M5. Although several solvers are compared, could comparable gains come from changing the backend of competing differentiable methods? Have alternative supported backends been explored, or the selected configurations shown to be competitive?
-
Failure modes. Please characterize when construction, retrieval, and storage costs outweigh avoided solves. The conditional efficiency analysis offers limited practical guidance on when FastDFL should help.
Questions for authors
-
Could the benchmark in Appendix D.1 be emphasized? Which modeling choices are necessary to fit the framework, and which define the testbeds? Fama is semi-synthetic, while Battery and M5 use particular downstream QP formulations.
-
Why is decision quality only “nearly unchanged” despite the exact-equivalence result? Appendix E.2 reports rare but material Jacobian discrepancies. How are these diagnosed, and could checks detect them?
-
Why use , rather than predicting the linear objective coefficient directly?
-
How do construction checks distinguish redundant zero-multiplier constraints from weakly active, nonredundant constraints at nondifferentiable points? What ensures differentiability without strict complementarity, and how is an affine multiplier map selected when dual solutions are nonunique?
-
A region-specific notation for the constant Jacobians in Eqs. (6),(7), or evaluation at , might improve clarity. This is a notation suggestion, not a mathematical objection.
-
Please separate the high-level reuse idea from construction/retrieval details more clearly, especially in Section 4.2. Measuring the quantities in the conditional efficiency analysis would also connect the bounds to observed behavior.
8 comments
Response to W3 and W4
To W3:
We separate the effects of prediction dimension, QP size and sample count on training time and storage, and have updated the QP scale and resource results in Appendix E.3, lines 1352–1356 and 1383–1395, and added the prediction dimension study in lines 1396–1403 of the revised manuscript.
For prediction dimension, we vary the cost coordinates while holding the QP structure fixed. The Battery formulation uses the original features and prices, with . Predictions specify coefficients of a fixed basis for costs across storage units and hours:
| Training speedup | Formula arrays (MiB) | |
|---|---|---|
| 12 | 2.10 | 143.0 |
| 24 | 1.76 | 336.4 |
| 36 | 1.62 | 482.5 |
| 48 | 1.53 | 643.2 |
| 96 | 1.34 | 938.6 |
Higher prediction dimension increases map storage, while FastDFL retains a time advantage through . At every fixed , mean speedup also increases at the recorded checkpoints from epoch 5 to 40:
For QP size, we fix and compare the following scales.
| Speedup, epoch 20 | Speedup, epoch 40 | Computation reduction, epoch 40 (%) | |
|---|---|---|---|
| (96,288) | 1.28 | 1.36 | 67.7 |
| (144,408) | 1.25 | 1.29 | 68.8 |
| (192,528) | 1.24 | 1.27 | 68.6 |
The time advantage increases at all three scales as training proceeds. By epoch 40, each scale replaces about 2/3 of solver and derivative computations.
For dataset size, we use subsets of the original 1,313-instance. This comparison measures both complete training time and the storage associated with individual samples.
| DAQP–KKT time (s) | FastDFL time (s) | Speedup | Formula arrays (MiB) | History bytes/sample | |
|---|---|---|---|---|---|
| 320 | 74.86 ± 2.46 | 53.35 ± 3.34 | 1.41 ± 0.07 | 56.1 | 104.3 |
| 640 | 124.25 ± 4.97 | 72.02 ± 2.35 | 1.73 ± 0.04 | 128.9 | 110.6 |
| 1280 | 215.93 ± 2.05 | 122.65 ± 1.03 | 1.76 ± 0.02 | 330.3 | 124.7 |
Storage for an individual sample is distinct from the shared formula library. Histories consume about 104 to 125 bytes per sample for references and containers. Native solver state buffers add about 6 KiB per sample in both methods. Shared formula arrays account for the larger library cost. The sample count experiment also measures the peak memory of the complete process. At , its means are 1170.1 MiB for DAQP–KKT and 1516.8 MiB for FastDFL. Together, the measurements expose the training time advantage of FastDFL, shared storage and smaller history cost for each sample, supporting a resource choice within the workload limit.
To W4:
We agree that backend choice can change the cost of differentiable optimization. We examine both solver selection and reuse with the solver held fixed. We evaluate the official dQP with OSQP and DAQP, retaining its original backward implementation, and test FastDFL with OSQP alongside its DAQP backend.
| Task | Backend | dQP time (s) | Backend–KKT time (s) | FastDFL time (s) |
|---|---|---|---|---|
| Fama | OSQP | |||
| Fama | DAQP | |||
| Battery | OSQP | |||
| Battery | DAQP | |||
| M5 | OSQP | |||
| M5 | DAQP |
It can be found that switching dQP to DAQP reduces its training time, while FastDFL remains faster with that same backend. To examine reuse directly, the OSQP–KKT comparison holds the forward solver and direct KKT backward computation fixed. Its mean speedups are 1.66, 1.26 and 2.55 times on Fama, Battery and M5. Solver selection and formula reuse address complementary costs. The former lowers the cost of each direct computation, and the latter reduces the number of direct computations required. We have added this comparison in Appendix E.4 (Table 12), lines 1427–1448 of the revised manuscript.
Response to W5
To W5:
For FastDFL, formula construction is an upfront investment, and subsequent uses determine its return. Consider a sequence involving 5 complete binding sets \textbf{ABCDE}. Each optimization layer evaluation includes a forward decision and a backward computation. Assume direct computation costs 1 s, lookup and validity checks cost 0.1 s per evaluation, and construction adds 0.5 s. Formula evaluation for the forward and backward computations costs 0.1 s, giving a total of 0.2 s for successful reuse. Then we have the following result:
| Successive evaluations | Direct computation (s) | FastDFL (s) | Stored formulas (used formulas) |
|---|---|---|---|
| AA | 2.0 | 2.7 | 1 (0) |
| AAAA | 4.0 | 0.8 | 1 (1) |
| BBCCDDEE | 8.0 | 10.8 | 5 (1) |
| Total | 14.0 | 14.3 | 5 (1) |
During AA, both evaluations require direct computation, and the second constructs A's formula. The next AAAA reuse it, bringing cumulative time to 3.5 s against 6 s for direct computation. Each of BCDE then appears twice, triggering construction without subsequent reuse. These evaluations add 2.8 s of overhead and erase the earlier 2.5 s saving. Here, 4/5 stored formulas supply no avoided computation. The second observation triggers construction, while subsequent uses determine its return. Recurrence saves time when retrieval and formula evaluation remain cheaper than direct computation.
Practical guidance is to assess subsequent formula use together with construction time, lookup time and memory growth. The QP formulation and data distribution affect the complete binding sets encountered. The training split, sample order and predictor updates affect recurrence and retrieval order. QP size and backend determine the relative costs of direct computation and reuse, while available memory limits library growth. Existing training records therefore support an assessment for the intended workload. More importantly, dynamically removing formulas with little subsequent use under a memory budget remains a promising future direction. We have added the cost insight to Section 6, lines 493–496, and the practical guidance to Appendix E.3, lines 1420–1424 of the revised manuscript.
Response to W1, W2&Q3
Dear Reviewer YBTR,
Thanks for your insightful comments! Below are our point-by-point responses:
To W1:
We agree that local affine solution maps and their constant Jacobians are established foundations. Our proposed FastDFL contributes the decisions governing formula construction and retrieval during DFL training. It constructs maps when constraint structures recur, uses each sample's formula history, and budgets searches of older formulas. We have clarified this contribution in Section 1, lines 58–60 of the revised manuscript.
These decisions exploit repeated visits to individual training samples. A sample can return to an earlier region after visiting other regions, allowing its history to recover a valid stored formula. A structure encountered only once can incur construction cost without subsequent reuse. Waiting for recurrence filters these constructions, while the search budget controls the cost of checking older formulas.
We add a regional MPC method for comparison. Regional MPC with active set updates (König and Mönnigmann, 2017) reuses the current region and updates active constraints between successive predictions. It constructs updated maps through the common KKT implementation and uses DAQP when an update is unavailable. The folllwing table shows the results of Regional MPC compared with FastDFL.
On Fama and M5, Regional MPC avoids more complete solves, while repeated map construction requires more processing. On the Battery task, unavailable regional updates also lead to direct solves. Actually, the ablation study in our original manuscript complements this comparison by identifying the construction and retrieval policies' effects. We have added Regional MPC to Section 5.2 (Table 1), lines 419–426, and Appendix E.1 (Table 9), lines 1261–1268 of the revised manuscript.
To W2 and Q3:
The fixed Hessian and feasible set define the present framework's scope. FastDFL targets prediction of cost vectors for a fixed decision problem, including returns under a fixed portfolio risk model and prices under fixed storage dynamics and limits. These assumptions fix the local KKT matrix, allowing reuse of the decision map and Jacobian when constraint structures recur.
The form q0+Qa also includes direct prediction of the complete linear coefficient vector. Setting q0=0 and Q=I (identity matrix) gives q(a)=a. The more general form expresses known task relationships. In the Battery task, a contains 24 hourly prices, Q places each price in the power coefficients of both storage units, and q0 supplies fixed coefficients. This representation keeps duplicate coefficients and known constants in the decision model.
The structural requirement for exact affine reuse is a fixed reduced KKT matrix with an affine on the right hand side. Fixed P,A,G with affine q(a),b(a),h(a) can retain that structure, subject to corresponding construction and validity checks. General predictions of Hessians or constraint matrices change the KKT matrix and require different local maps, checks or approximations. We identify these extensions as open issues in the Discussion.