acceptodds
Under review as a conference paper at ICLR 2027

Cost-Aware Active Learning of Optimal Control Policies on the Switching Set

Abstract

We consider discounted optimal control of control-affine systems under input constraints. The optimal feedback of such problems saturates and switches between input levels across surfaces in the state space. Imitating such a controller requires one optimal control solve per expert sample, so the samples must be placed where they matter. For the policy over a finite input grid, learned as a classifier, we show that these samples should lie on the switching surfaces. Their value there is set by the cost slope g, the rate at which the cost of the wrong input grows with the distance from the surface. We prove that the cost-sensitive loss of the learned policy is the -weighted squared error of its switching surfaces and that the optimal sample density on them is proportional to , with the local rate of the learner. Margin sampling loses a constant factor, computable from the expert before learning, and an idealised learner improves the loss rate of uniform sampling from to in state dimensions. The practical sampler estimates from the already-returned action values, with no additional expert samples. On the inverted pendulum, two mountain-car configurations and a four-dimensional cart-pole, the sampler matches the accuracy of uniform sampling with to more than times fewer expert samples and its closed-loop cost gap with to times fewer. The factor computed beforehand predicts on which systems the cost weighting pays off.

Primary area:
reinforcement learning
Keywords:
active learning, imitation learning, cost-sensitive classification, constrained optimal control, switching-set active learning, explicit model predictive control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nV9BOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to allocate a limited expert-query budget when learning constrained optimal control policies on a finite input grid. It proposes concentrating queries near switching surfaces and weighting their allocation by the local cost slope. Under boundary-fragment and linear cost-growth assumptions, the analysis relates cost-sensitive loss to weighted squared boundary error and derives allocation rules for an idealised learner. The practical method uses a neural classifier and estimates slopes from previously returned advantage rows.

Critical strengths
  1. Relevant problem and a useful geometric perspective. The paper addresses an important question in policy imitation with expensive experts: where should a limited query budget be spent? Connecting switching-set geometry with cost-sensitive active learning gives the sampling rule a clear interpretation grounded in control structure.

  2. An informative relationship between boundary error and control cost. Proposition 1 distinguishes classification disagreement, which corresponds to boundary L1L^1 error, from cost-sensitive loss, which is bounded by slope-weighted squared L2L^2 boundary error. This helps explain why similar classification accuracy can correspond to different control costs and why the two objectives may favour different allocations.

  3. A control-theoretic interpretation of the cost slope. Lemma 2 connects action-value differences with the switching function, providing a local interpretation of how input jumps and switching-function variation affect the cost of an incorrect action. This offers an intuitive motivation for distinguishing switches between neighbouring levels from large input reversals.

  4. Simple and interpretable allocation rules. For the idealised learner, minimising the error bound yields the Neyman allocation that combines surface size and cost slope. Corollary 5 further connects the allocation exponent to the learner's local error rate. These results provide a useful framework for reasoning about allocation choices, with qualitative support from the experimental comparisons.

  5. Useful ablations and reporting of negative results. The experiments examine alternative samplers, slope sources, allocation exponents, coverage, and model capacity. They also report unsuccessful occupancy weighting and differences between local and global learners. These comparisons help identify the sources and limitations of the observed gains.

Critical weaknesses
  1. The analysis has limited coverage of the practical algorithm. Theorem 3 explicitly concerns an idealised bisection learner, whereas Algorithm 1 combines a global neural classifier, sampling from a low-margin band, estimated slopes, and batch retraining. The stated rates and allocations do not directly cover these components. Corollary 5 assumes a local error-rate model that has not been established for the practical network, and the global classifier in Appendix E favours flatter allocations than the local learner. The theory therefore supports a sampling design principle, while the label-complexity implications for Algorithm 1 remain unclear.

  2. The closed-loop bound provides limited practical guidance. Theorem 6 depends on a policy-specific concentrability coefficient CπC_\pi, which may be large and is not further controlled for the learned policies. Consequently, the cost-sensitive analysis does not yet yield a useful quantitative guarantee on closed-loop performance. The authors acknowledge that the metrics can rank learners differently, and the experiments show that reductions in cost-sensitive loss do not consistently translate into comparable closed-loop gains. The bound offers a conceptual connection, but its ability to guide sampling for deployment remains limited.

  3. Query efficiency depends on a demanding expert-feedback model. Slope estimation requires a full action-value or advantage row at each queried state. This is readily available from an expert implemented as a converged semi-Lagrangian table, but obtaining it from an optimisation-based expert may cost substantially more than obtaining the optimal action alone. Although this limitation is acknowledged, the reported query savings do not establish corresponding computational savings. The proposed pre-learning gap statistic also relies on the reference semi-Lagrangian table. Its acquisition cost and reliability under a limited query budget remain unclear.

  4. The consistency claim without importance weights needs qualification. Lemma 7 provides a risk comparison and characterises zero-risk policies under the sampling distribution. These results support a population-level argument when the policy class contains an optimal policy, but do not establish that the best approximation within a restricted neural-network class is independent of the sampling distribution. With approximation error, the minimiser generally depends on that distribution. A4 does not establish that the practical network can represent an optimal policy, and Lemma 7 does not establish consistency of its empirical training procedure under adaptive sampling. The stronger wording in Section 3 should be qualified accordingly.

  5. The geometric assumptions need a clearer connection to concrete control problems. A1 and A2 require separated boundary fragments and linear cost growth with strictly positive slope bounds. Junctions of switching surfaces, degenerate switching, and insufficient value-function regularity can restrict their applicability. The paper already limits its rate-optimality claim to the boundary-fragment class. What remains insufficiently developed is a clear account of control problem subclasses for which the assumptions can be verified, and of their applicability to the reported benchmarks.

  6. The additional closed-loop benefit of the slope-estimate sampler is system dependent. The experiments provide substantial support for switching-set sampling over uniform labelling, but the additional closed-loop benefit of the slope-estimate sampler over other switching-set samplers is less consistent. It performs strongly in some settings and is comparable to, or outperformed by, alternatives in others. Its benefit is clearer for cost-sensitive loss. The practical contribution would be more precisely characterised by distinguishing the general benefit of switching-set sampling from the additional benefit of estimating slopes.

Questions for authors
  1. Connection between the theory and the practical learner. What analysis or empirical evidence supports applying the local rate a=na=n from Corollary 5 to the neural classifier? Given the flatter allocations preferred by the global network in Figure 3, how should the choice β=1/2\beta=1/2 and its range of applicability be understood?

  2. Usefulness of the closed-loop guarantee. Can the authors provide estimates, bounds, or verifiable sufficient conditions for CπC_\pi in the benchmark systems, so that the practical strength of Theorem 6 for the learned policies can be assessed?

  3. Expert computation and the pre-learning gap statistic. For an optimisation-based expert, what additional computation is needed to return a full advantage row rather than an optimal action? How do the reported query savings translate into computational savings? Without the reference semi-Lagrangian table, can the gap statistic be estimated from a coarse model or pilot queries, and how would their cost and estimation error be accounted for?

  4. Conditions for training without importance weights. Should the claims of a common population minimiser and consistency of unweighted fitting be restricted to policy classes that can represent an optimal policy? How do the authors account for the dependence of the best approximation on the sampling distribution when the network has approximation error?

  5. Verification of A1 and A2. Can the authors identify a concrete control problem subclass or representative example in which A1 and A2 can be verified? Which regions or features of the benchmark systems are covered by these assumptions?

  6. The independent closed-loop benefit of the slope-estimate sampler. Under matched query budgets, training losses, and appropriately controlled allocation exponents, in which systems and budget regimes is the closed-loop advantage of the slope-estimate sampler over margin and jump sampling statistically supported? How should the results where alternatives perform comparably or better delimit the claimed benefit?

1 comment

We thank the reviewer for the detailed report. We agree that the claims should be narrowed. The rate improvement belongs to the idealised learner. Theorem 6 gives no quantitative closed-loop guarantee. We claim query savings and no computational savings. The statement on importance weights holds for policy classes that contain an optimal policy. The closed-loop benefit of the slope estimate beyond switching-set sampling is system-dependent. The next version of the paper will be revised accordingly.

Reviewer_fE7FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper proposes an active learning method for optimal control policies that concentrates expert queries near switching surfaces and weights them by the cost of choosing the wrong action.

Critical strengths

the derived sampling rule is well motivated and the comparisons with ordinary margin sampling give actionable insight onto when cost weighting actually matters.

Critical weaknesses

i am less convinced by the connection between the theoretical rates and the implemented algorithm. the improved rate is established for an idealised bisection learner while the practical sampler has a constant factor gain under the stated analysis.

Questions for authors

how much of the reported advantage would remain with an optimisation based expert, accounting for the additional solves needed ? how well does this method also scale to higher dimensional systems where it may be more expensive to find or sample near switching surfaces ? moreover could the authors also address the crucial weakness section

1 comment

We thank the reviewer for the careful reading and for the questions.

Questions. Regarding an optimisation-based expert, all strategies in our comparisons, including uniform labelling, use the full row of action values. Therefore, the reported query ratios against uniform labelling are also ratios of solves, regardless of the cost of a row. Switching-set sampling also works with the optimal action alone. With a cross-entropy loss, it maintains its gain in agreement over uniform labelling in all three two-dimensional configurations. This part of the advantage needs only one solve per query. The additional gain of the slope estimate is smaller and depends on the cost of the row. We have not evaluated the slope estimate with an optimisation-based expert and cannot claim any computational savings.

Regarding higher-dimensional systems, finding the band requires only forward passes of the classifier and no expert queries. This also holds in four dimensions. What decreases with the dimension is the gain. The query density in the band is 9.19.1 times the density under uniform labelling. This gives a constant factor of 9.19.1 in queries and of 9.12/n9.1^{2/n} in the loss, that is 9.19.1, 3.03.0 and 2.12.1 for n=2n=2, 44 and 66. Even the idealised learner only improves the rate from N−2/nN^{-2/n} to N−2/(n−1)N^{-2/(n-1)}, and these two rates converge as nn increases. On the four-dimensional cart-pole, the sampler requires 1.91.9 times fewer queries than uniform labelling to achieve the same agreement and 2.12.1 times fewer to achieve the same closed-loop cost gap. We have no evidence for more than four dimensions. Such systems are outside the scope of the paper, which focuses on the analysis. They require an approximate expert, whose label errors introduce additional effects that the analysis does not cover.

Weakness. We agree on the weakness. The improved rate belongs to an unimplemented bisection learner. Algorithm 1 fills a fixed-volume band and Corollary 5 provides it with a uniform labelling rate and constant factor gain. The learning curves support this. The sampler's curve is steeper than that of uniform labelling only at small budgets.

Reviewer_BTcGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies cost-aware active learning for learning optimal control policies under constrained control, where each expert query requires solving an expensive optimal control problem. The key observation is that most policy errors occur near the switching surfaces where the optimal action changes, but different parts of these surfaces have very different consequences when misclassified. The authors so propose to concentrate queries near the switching set and further weight them according to a locally estimated cost slope, which measures how quickly the penalty of choosing the wrong action increases away from the boundary. The paper provides a theoretical analysis relating policy loss to errors in the estimated switching surfaces and derives an optimal way of allocating samples according to this cost information. This shows an improved label-complexity rate over uniform sampling for an idealized learner. Based on this analysis, the authors develop a batch active-learning sampler that estimates the cost slope using action values already returned by the expert without requiring additional expert queries. The experiments show that the proposed sampler can achieve comparable policy accuracy and closed-loop performance with substantially fewer expert queries than uniform sampling.

Critical strengths
  1. The geometric framing is clean and useful.

  2. The slope estimate costs nothing extra with a table expert, since the rows are already returned, and the paper shows that it is what carries the gain.

  3. The paper is open about what it does not do. The limitations paragraph says that the theorem is for a bisection learner that is not implemented, that all experts are tables, that an optimisation-based expert returns the row only with more solves, and that the closed-loop statement depends on a concentrability constant that can be large.

Critical weaknesses
  1. The theoretical guarantees do not directly cover the implemented algorithm. The improved label-complexity rate is established for an idealized bisection learner, but the practical method samples a band around a neural classifier’s boundary.

  2. The method assumes that each query returns exact action values for every available input, which the experiments obtain from a precomputed table. However, the authors acknowledge that an optimization-based expert may require additional solves to provide these values and may introduce errors not covered by the analysis.

  3. Theorem 6 bounds the closed-loop cost gap using a policy-dependent concentrability coefficient, which can be large. So improving the cost-sensitive classification loss does not necessarily produce a comparable improvement in deployed control performance. The stronger surface-weighted characterization requires additional assumptions on state visitation, and the occupation-weighting experiment does not support using that characterization as a practical sampling rule. This leaves a gap between optimizing the proposed learning objective and guaranteeing better closed-loop behavior.

  4. The experimental scope is narrow to establish scalability and broad applicability.

Questions for authors
  1. Which mechanism do you believe produces the steeper learning curves of the band samplers? Corollary 5 predicts the uniform rate with a smaller prefactor for a band of fixed volume fraction τ\tau. Does the gain shrink to the volume ratio at large budgets, and can you show the local slopes of the learning curves against the budget?

  2. Is Theorem 6 ever informative here? For a deterministic closed loop that converges to an equilibrium the occupation measure concentrates and CπC_\pi can be infinite.

1 comment

We thank the reviewer for the careful reading. Both questions can be answered with the data from the experiments in Section 5.

Q1. Your reading of Corollary 5 is correct. Algorithm 1 fills a band of fixed volume fraction τ=0.1\tau=0.1, so it has the rate of uniform labelling and a constant gain. With the coverage share α=0.1\alpha=0.1, the query density in the band is 9.19.1 times the uniform one, which gives a factor of 9.19.1 in queries and 9.12/n9.1^{2/n} in the loss. The steeper curves are the transient towards that constant. Early on, a poor classifier places the band, and the sampler has no gain.

The local log-log slopes of the cost-sensitive loss support this. Uniform labelling has slopes of −0.8-0.8 to −1.1-1.1 in two dimensions (predicted −2/n=−1-2/n=-1). Between 128128 and 512512 queries the sampler is steeper by 0.380.38, 0.970.97, 0.390.39 and 0.230.23 (pendulum, mountain car with wu=0.001w_u=0.001, mountain car with wu=0.1w_u=0.1, cart-pole). The 95% bootstrap interval of this difference excludes zero on three of the four. In the last half of the budget, it includes zero on all four. The loss ratio at equal budget rises from about one to 5.05.0, 13.113.1 and 3.43.4 in two dimensions (constant 9.19.1) and to 1.71.7 on the cart-pole (constant 3.03.0), so the constant is matched in order of magnitude only. The ratio has levelled off on the mountain car with wu=0.001w_u=0.001 and still rises at 20162016 queries on the other two. A rate gain requires a band that shrinks as the budget shrinks. This is what the bisection learner does, and it is not implemented.

Q2. You are right that the bound is not informative on our benchmarks. A stabilising closed loop concentrates the occupation measure, and we do not claim a finite CπC_\pi for any learned policy. What remains useful is the identity in the same theorem, R(π)=Ceff(π)L(π)/(1−γh)R(\pi)=C_{\mathrm{eff}}(\pi) L(\pi)/(1-\gamma_h), where CeffC_{\mathrm{eff}} is the ratio of the advantage integrated under the occupation measure to that under ν\nu. It needs no assumptions and can be measured, since every run records RR and LL. The median of CeffC_{\mathrm{eff}} over seeds lies between 0.070.07 and 0.630.63, except for the pendulum under uniform labelling at up to 992992 queries (1.01.0 to 1.71.7). It varies with budget and sampler, and the rank correlation between LL and RR at a fixed budget lies between 0.280.28 and 0.860.86. The cost-sensitive loss is an informative and imperfect surrogate, and the analysis gives no quantitative closed-loop guarantee.

Weaknesses. We would like to thank the reviewer for pointing out these weaknesses. We will try to resolve them in the next version of the paper. The narrow experimental scope is a deliberate choice. We have restricted the experiments to systems in which a table serves as an exact expert, so that the analysis can be tested without the label errors of an approximate expert. Systems with more than four state dimensions are left for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.