- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to allocate a limited expert-query budget when learning constrained optimal control policies on a finite input grid. It proposes concentrating queries near switching surfaces and weighting their allocation by the local cost slope. Under boundary-fragment and linear cost-growth assumptions, the analysis relates cost-sensitive loss to weighted squared boundary error and derives allocation rules for an idealised learner. The practical method uses a neural classifier and estimates slopes from previously returned advantage rows.
Critical strengths
-
Relevant problem and a useful geometric perspective. The paper addresses an important question in policy imitation with expensive experts: where should a limited query budget be spent? Connecting switching-set geometry with cost-sensitive active learning gives the sampling rule a clear interpretation grounded in control structure.
-
An informative relationship between boundary error and control cost. Proposition 1 distinguishes classification disagreement, which corresponds to boundary error, from cost-sensitive loss, which is bounded by slope-weighted squared boundary error. This helps explain why similar classification accuracy can correspond to different control costs and why the two objectives may favour different allocations.
-
A control-theoretic interpretation of the cost slope. Lemma 2 connects action-value differences with the switching function, providing a local interpretation of how input jumps and switching-function variation affect the cost of an incorrect action. This offers an intuitive motivation for distinguishing switches between neighbouring levels from large input reversals.
-
Simple and interpretable allocation rules. For the idealised learner, minimising the error bound yields the Neyman allocation that combines surface size and cost slope. Corollary 5 further connects the allocation exponent to the learner's local error rate. These results provide a useful framework for reasoning about allocation choices, with qualitative support from the experimental comparisons.
-
Useful ablations and reporting of negative results. The experiments examine alternative samplers, slope sources, allocation exponents, coverage, and model capacity. They also report unsuccessful occupancy weighting and differences between local and global learners. These comparisons help identify the sources and limitations of the observed gains.
Critical weaknesses
-
The analysis has limited coverage of the practical algorithm. Theorem 3 explicitly concerns an idealised bisection learner, whereas Algorithm 1 combines a global neural classifier, sampling from a low-margin band, estimated slopes, and batch retraining. The stated rates and allocations do not directly cover these components. Corollary 5 assumes a local error-rate model that has not been established for the practical network, and the global classifier in Appendix E favours flatter allocations than the local learner. The theory therefore supports a sampling design principle, while the label-complexity implications for Algorithm 1 remain unclear.
-
The closed-loop bound provides limited practical guidance. Theorem 6 depends on a policy-specific concentrability coefficient , which may be large and is not further controlled for the learned policies. Consequently, the cost-sensitive analysis does not yet yield a useful quantitative guarantee on closed-loop performance. The authors acknowledge that the metrics can rank learners differently, and the experiments show that reductions in cost-sensitive loss do not consistently translate into comparable closed-loop gains. The bound offers a conceptual connection, but its ability to guide sampling for deployment remains limited.
-
Query efficiency depends on a demanding expert-feedback model. Slope estimation requires a full action-value or advantage row at each queried state. This is readily available from an expert implemented as a converged semi-Lagrangian table, but obtaining it from an optimisation-based expert may cost substantially more than obtaining the optimal action alone. Although this limitation is acknowledged, the reported query savings do not establish corresponding computational savings. The proposed pre-learning gap statistic also relies on the reference semi-Lagrangian table. Its acquisition cost and reliability under a limited query budget remain unclear.
-
The consistency claim without importance weights needs qualification. Lemma 7 provides a risk comparison and characterises zero-risk policies under the sampling distribution. These results support a population-level argument when the policy class contains an optimal policy, but do not establish that the best approximation within a restricted neural-network class is independent of the sampling distribution. With approximation error, the minimiser generally depends on that distribution. A4 does not establish that the practical network can represent an optimal policy, and Lemma 7 does not establish consistency of its empirical training procedure under adaptive sampling. The stronger wording in Section 3 should be qualified accordingly.
-
The geometric assumptions need a clearer connection to concrete control problems. A1 and A2 require separated boundary fragments and linear cost growth with strictly positive slope bounds. Junctions of switching surfaces, degenerate switching, and insufficient value-function regularity can restrict their applicability. The paper already limits its rate-optimality claim to the boundary-fragment class. What remains insufficiently developed is a clear account of control problem subclasses for which the assumptions can be verified, and of their applicability to the reported benchmarks.
-
The additional closed-loop benefit of the slope-estimate sampler is system dependent. The experiments provide substantial support for switching-set sampling over uniform labelling, but the additional closed-loop benefit of the slope-estimate sampler over other switching-set samplers is less consistent. It performs strongly in some settings and is comparable to, or outperformed by, alternatives in others. Its benefit is clearer for cost-sensitive loss. The practical contribution would be more precisely characterised by distinguishing the general benefit of switching-set sampling from the additional benefit of estimating slopes.
Questions for authors
-
Connection between the theory and the practical learner. What analysis or empirical evidence supports applying the local rate from Corollary 5 to the neural classifier? Given the flatter allocations preferred by the global network in Figure 3, how should the choice and its range of applicability be understood?
-
Usefulness of the closed-loop guarantee. Can the authors provide estimates, bounds, or verifiable sufficient conditions for in the benchmark systems, so that the practical strength of Theorem 6 for the learned policies can be assessed?
-
Expert computation and the pre-learning gap statistic. For an optimisation-based expert, what additional computation is needed to return a full advantage row rather than an optimal action? How do the reported query savings translate into computational savings? Without the reference semi-Lagrangian table, can the gap statistic be estimated from a coarse model or pilot queries, and how would their cost and estimation error be accounted for?
-
Conditions for training without importance weights. Should the claims of a common population minimiser and consistency of unweighted fitting be restricted to policy classes that can represent an optimal policy? How do the authors account for the dependence of the best approximation on the sampling distribution when the network has approximation error?
-
Verification of A1 and A2. Can the authors identify a concrete control problem subclass or representative example in which A1 and A2 can be verified? Which regions or features of the benchmark systems are covered by these assumptions?
-
The independent closed-loop benefit of the slope-estimate sampler. Under matched query budgets, training losses, and appropriately controlled allocation exponents, in which systems and budget regimes is the closed-loop advantage of the slope-estimate sampler over margin and jump sampling statistically supported? How should the results where alternatives perform comparably or better delimit the claimed benefit?
We thank the reviewer for the detailed report. We agree that the claims should be narrowed. The rate improvement belongs to the idealised learner. Theorem 6 gives no quantitative closed-loop guarantee. We claim query savings and no computational savings. The statement on importance weights holds for policy classes that contain an optimal policy. The closed-loop benefit of the slope estimate beyond switching-set sampling is system-dependent. The next version of the paper will be revised accordingly.