acceptodds
Under review as a conference paper at ICLR 2027

Calibrating Verified Task Sets with Limited Measurements

Abstract

A learned controller can meet a task specification on several disconnected regions of physical parameter space. With few calibration measurements, a generic confidence interval may certify none of those regions. We study how a verified task set can instead determine a fixed-task acceptance rule. For a scalar Gaussian estimate, we reduce the worst unsafe-parameter risk to finitely many boundary evaluations when every acceptance component is separated from its boundary by at least one standard error. The result supports component-specific radii; maximizing a same-component power objective then becomes a convex optimization problem. Exact-rational inner-set verification connects the statistical calculation to repeated feedback with a shared uncertain restitution coefficient. On 63 tasks from 15 learned controllers, four-measurement acceptance increases from 41.0% with a gap-based test to 44.0% with a common boundary-calibrated radius and 44.7% with component allocation. Eight tasks change from zero to positive acceptance. Tighter-tolerance experiments locate this benefit in the narrow-component regime, and an independent checkpoint study compares calibration-aware selection with imitation and task-loss selection. Matched composite-null, continuum-search, and conformal controls identify which gains require Gaussian information and which are computational. The contribution is a calibrated post-training validation procedure under explicit model and measurement contracts, rather than a guarantee of unrestricted physical transfer.

Primary area:
applications to robotics, autonomy, planning
Keywords:
controller verification, task-set certification, learning-based control

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_S4PvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I think the paper wants to measure if a trained policy, like a robot, can be deployed successfully without errors. But it could be a lot of different things, it is quite unclear reading the paper what the problem is.

Critical strengths

I am not sure if there is one.

Critical weaknesses

I would love to just be able to read this paper, it makes no sense to me.

Questions for authors

Why submit a paper in this state, this is irrespecutful of the reviewers' time.

1 comment
AuthorsOpenReview ↗

Response to Reviewer S4Pv

Thank you for the frank feedback. We rewrote the abstract in plain words and added a worked example to the introduction (new Figure 1(a), page 2).

The problem. A neural-network controller drives an object between two walls toward a goal. Whether it succeeds depends on the restitution (the fraction of speed the object keeps at each bounce), which is not known exactly. A verifier proves that the task succeeds for every restitution in a set S, so S contains no failing value. For learned controllers S often consists of several separate intervals (five in Figure 1(a)). We can afford only a few noisy measurements of the restitution, for example four. When does the resulting estimate justify claiming success, so that a wrong claim has probability at most 5% whatever the true value is?

Why the standard answers fail. The usual rule claims success only when the 95% confidence interval (a plausible range for the restitution) fits inside one safe interval, which never happens in Figure 1(a) because every safe interval there is too short. Applying a 5% rule to each safe interval separately is invalid. If the true restitution is a failing value in a short gap between two safe intervals, the rules on both sides can each wrongly claim success, together with probability up to 9.8% (Section 5.2). A valid alternative, the standard "gap rule", claims success only when the estimate lies inside a safe interval by a margin that grows as the narrowest gap between safe intervals shrinks. In Figure 1(a) it also never claims success.

What the paper contributes. With K safe intervals, Theorem 1 shows that the worst probability of a wrong claim over the infinitely many failing values equals the largest of 2K exactly computable numbers. This holds whenever every estimate that triggers a claim lies at least one standard error (the estimate's typical error) inside its interval. With a separate margin for each interval, Theorem 2 shows that choosing the margins to maximize a lower bound on the chance of a correct claim is a convex optimization problem, which can be solved reliably. Averaged over 63 tasks (different controllers, starts, goals and horizons) and over true values in each verified set, a correct claim with four measurements is made 41.0% of the time with the gap rule, 44.0% with our one-margin rule and 44.7% with our per-interval margins. All three rules keep the probability of a wrong claim at or below 5%.

If any passage is still unclear, we would be grateful for a pointer to it.

Reviewer_Vp3sOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses verification of controllers that have disconnected safe intervals i.e. intervals of a given parameter where the controller is verified to work. The main issue is that standard approaches such as a confidence interval or a TOST do not handle disconnected safe intervals well leading to either no certificates of any interval or too many false certificates. The authors show that limited measurements are enough for calibrating the claim of the controller being safe and identify a convex optimization problem to learn where to allow certificates. Additionally, they investigate to which controllers the certifiability with limited measurements applies and how choosing a controller with a imitation-based loss compares to choosing one by anticipated certification performance.

Critical strengths

S1: The paper states the limits of its contributions well

Critical weaknesses

W1: The writing is severely lacking clarity. The central research gap could be stated more clearly and the explanations of the theoretical results and their consequences seem unclear to me. The figures and tables are not explained well and the structure of the experiments could be outlined more clearly. W2: The experiments are done on a single system. The system seems very limited in its complexity and therefore limits the conclusions that can be drawn from the experimental results.

Questions for authors
  • The experiments are done on a single system. How would the experiments scale to more systems and in particular more complex systems?
  • What are the 63 tasks described in the experimental section?
  • You mention the task cost selector and the certificate utility selector differ in only 2 seeds. Why could there be such a big overlap?
1 comment
AuthorsOpenReview ↗

Response to Reviewer Vp3s

Thank you for the accurate summary and concrete questions. We agree that the writing was too compressed.

W1: clarity.

  • Research gap. The first paragraph of Section 1 now works through a concrete task (Figure 1(a), page 2) and states the question in one sentence. It then explains why, when the safe intervals or the gaps between them are narrow, a confidence interval may certify nothing and per-interval tests either exceed the 5% level or certify nothing.
  • Theory. Each theorem in Section 3 is now followed by its consequence. By Theorem 1, a rule's validity is checked exactly with 2K numbers, two per safe interval, instead of a search over infinitely many unsafe values. By Theorem 2, choosing where to accept is a convex problem in which the risk bound enters as constraints.
  • Figures and tables. The captions of Figure 1 and Tables 2 to 5 now define every rule and column.
  • Experiments. Section 5 opens with a roadmap that says what each of Sections 5.1 to 5.5 shows.

Q2: what are the 63 tasks? A task fixes a trained controller, a start position, a goal, a horizon H of 12, 24 or 48 steps, and a tolerance of 0.1 on the final distance to the goal. The 15 controllers are ReLU networks with eight hidden units, trained to imitate a fixed nonlinear feedback law. Three controllers give nine tasks each (three start and goal pairs per horizon) and the other twelve give one per horizon, 63 in total. No task was discarded because of the shape of its verified set (Section 5, Appendix E).

W2 and Q1: a single, simple system, and scaling. We added a planar system with four walls, one shared restitution and a controller that couples the two axes, checked by a new exact verifier (Section 5.5, Table 5, Appendix H). There we compare our common-radius rule, which uses one acceptance margin for all intervals calibrated with Theorem 1, with the gap rule, the standard equivalence test with its margin widened for the narrowest unsafe gap. With four measurements, on the 55 planar tasks with a nonempty verified set, our rule accepts 2.30 percentage points more often (95% interval [1.74, 2.94]). Part of this gain is classical: 0.71 of the 2.30 points come from tasks with one safe interval, where our rule reduces to a standard equivalence test. Since both systems share the contact model, the planar system tests new shapes of the verified set, not new physics.

Larger states, longer horizons and bigger controllers affect only the verification step. Its median cost per task is 14.8 s for the planar system at H = 24, against 0.70 s in one dimension, where it grows to 13.75 s at H = 48 (Table 12). Other systems, such as a robot arm, need their own sound verifier, which we have not demonstrated. The real limit is the number of uncertain parameters, because Theorem 1 already fails with two (Appendix D.3).

Q3: why do the task-cost and certificate selectors overlap? They overlap mostly because both reward accurate final positions, and across checkpoints their scores have a Spearman correlation of 0.73 (Appendix I, Table 14). The small pool of four checkpoints is not the cause, since doubling it leaves their disagreement unchanged (10 of 32 runs). They chose differently in 2 of the 8 submitted runs (Figure 2, now in Appendix I) and in 22 of all 64 runs (Section 5.3). Certification-based selection gives 1.19 points higher held-out certificate utility on average (95% interval [0.48, 2.09]), and it is better in 19 runs and worse in 1. Its advantage over selecting by simulated success rate, however, is only 0.38 points, because both criteria count whether the final position is within the tolerance.

Reviewer_QxNqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops finite-boundary risk calibration and component allocation for verified task sets under scalar Gaussian measurements, connecting statistical acceptance rules to learned-controller verification. It provides theoretical analysis, but the experimental validation remains overly simplistic and entirely synthetic.

Critical strengths

Explicit theoretical derivations and proofs, together with independent numerical risk checks, provide a clear mathematical foundation.

Critical weaknesses

Validation is limited to an extremely simple one-dimensional system and synthetic measurements. The learning-related contribution remains modest, with limited improvement over task-cost checkpoint selection.

Questions for authors
  1. Which parts of Theorems 1 and 2 go beyond existing equivalence tests and composite-null testing methods? Please distinguish the contributions of finite-boundary reduction, heterogeneous radii, and convex allocation from established results, including their computational advantages.
  2. Theorem 2 establishes convex optimization for same-component power Q, whereas the main allocated results optimize full power, which need not be concave. Please report both formulations, sensitivity to initialization, and fallback frequency, and restrict optimality claims accordingly.
  3. How do model mismatch, measurement bias, heteroscedasticity, and non-Gaussian errors affect validity? The existing robustness controls are useful but synthetic. Could real measurements or explicit mismatch bounds substantiate the practical measurement assumptions?
  4. What learning-specific benefit does the method provide beyond generic post-training statistical validation? Please clarify controller architectures, training data, checkpoint schedules, and data splits, and assess whether the small improvement over task-cost selection is reliable across independent seeds.
  5. Your counterexample rules out a direct multivariate boundary extension. What are the computational costs of robust scalar profiles or full unsafe-set covers? How does the implementation maintain validity under adaptive verification refinement or sequential data collection?
3 comments
AuthorsOpenReview ↗

Response to Reviewer QxNq (1/2)

Thank you for the detailed review and for crediting the derivations and checks.

W1: one simple system and synthetic data. We added a second verified system with two coupled axes (Section 5.5), explicit mismatch bounds (Q3) and a 64-seed checkpoint study (Q4). On the planar system the gain over the gap rule recurs, at 2.30 percentage points with four measurements (95% interval [1.74, 2.94]). However, 0.71 of these points come from single-interval tasks, where the gain is the classical equivalence-testing effect. On the one-dimensional system those tasks gain nothing, so all of its 2.99-point gain comes from tasks with several intervals.

Q1: what goes beyond equivalence and composite-null testing? For a single interval our rule is classical, and we do not claim it as new. Its common radius is that of Romano's uniformly most powerful equivalence test (2005, Example 1.1), and the two radii agree numerically to 1.3 × 10⁻¹⁵ for intervals at least 2.5 standard errors wide (Appendix J.1). What is new concerns K > 1 intervals, where an unsafe gap receives acceptance from both neighbors.

  • Theorem 1 reduces the supremum over the continuum of unsafe parameters to the largest of 2K endpoint values, provided accepted estimates stay one standard error inside their interval. Without this condition the maximum can lie inside a gap (Appendix A.2), and with two parameters inside the unsafe region (Appendix D.3).
  • The reduction holds for different radii in different intervals, whereas the classical constructions we compare against use one radius for every interval.
  • Theorem 2 shows that the boundary constraints are convex and that the same-component power Q is concave.
  • Corollary 5 extends these results to a class of known non-Gaussian error laws (see Q3).

Combining classical single-interval tests is either invalid or less powerful (Appendix J.1, Table 17). Testing each interval separately at level δ reaches a maximum risk of 9.79%, and at four measurements valid Bonferroni splits of δ across intervals accept 2.3 to 7.4 points less often than our common radius. Park and Wang's pointwise-rejection test yields our common-radius rules if its approximate adjustment is replaced by an exact calculation over all unsafe parameters (Section 5.2). The pointwise principle is therefore theirs, and our contribution is the exact finite maximum and its allocation.

The computational advantage is a constant factor, not a better asymptotic order. A complete boundary calibration takes a median of 1.9 ms, against 0.17 s for the rigorous continuum search of Section 3.2 (a median ratio of 72; Appendix J.2). In both, the number of distribution-function evaluations grows roughly quadratically in K. The boundary route also gives the exact risk rather than a tolerance-limited bound.

Q2: Q versus full power. We now report both objectives on all 612 combinations of task and measurement budget (Appendix F, Table 8). Mean acceptance is identical to one decimal place for every cohort and budget, and the largest difference for any task is 1.1 × 10⁻⁴ points. Over 20 random feasible starts plus the common-radius start, the final full power varies by at most 1.2 × 10⁻⁷ points. The fallback to the common rule occurs in 26 of the 612 combinations, exactly those in which no interval is wide enough to optimize (Appendix F). Section 3.1 claims global optimality only for Q, and validity never depends on the optimizer, because every returned rule passes an independent risk check.

AuthorsOpenReview ↗

Response to Reviewer QxNq (2/2)

Q3: mismatch, bias, heteroscedasticity and non-Gaussian errors. For rules whose radii are at least one standard error, Proposition 4 and Corollary 5 (Section 4.2, proofs in Appendix G) replace each of these assumptions by an explicit margin.

  • A bias of at most b̄ and a noise scale of at most s̄, both possibly varying with the parameter, are handled by shrinking each interval of S by b̄ at both ends and calibrating at scale s̄. This covers heteroscedastic noise, but not bias that depends on the noise itself.
  • Theorems 1 and 2 hold exactly for known symmetric, unimodal error laws whose density is convex beyond one standard error, such as the Laplace, logistic and Student t laws.
  • An error law within Kolmogorov distance κ of the normal law needs level δ − 2Kκ, which is cheap only for very small κ.
  • A model error of at most η in the final state is handled by verifying at tolerance ε − η. This margin is costly: on the 30 tasks of Table 3 with four measurements and the prior kept on the tolerance-0.1 set, verifying at 0.05 instead of 0.1 (η = 0.05) lowers the acceptance of our allocated rule from 46.3% to 10.3%.
  • Margins estimated from data must have their failure probability subtracted from δ.

With four measurements (Appendix G, Table 9), an uncorrected bias of 0.25 standard errors raises the maximum risk to 8.15%, and the corrected rule accepts 40.0% instead of 44.0%. If the noise scale rises linearly from 0.75s̄ to s̄ across the domain, a rule calibrated at the mean scale reaches 7.50% risk, while calibrating at s̄ keeps the rule valid and accepts 40.1% instead of 44.0%. A rule calibrated for five error laws at once costs only 0.16 to 0.25 points.

Lacking real measurements, we tested the measurement model on rebound data simulated with the MuJoCo physics engine (Appendix G.4). There, restitution rises with approach speed, so the slope that eq. (11) estimates from rebound and approach speeds is not the restitution the model needs, and no residual test can detect this bias. If the model needs the restitution at the lower approach speed, the uncorrected risk is 7.7% to 11.2% with four measurements. A bias bound from independent data makes the measurement rule valid again. Because speed-dependent restitution also violates the model's constant-restitution dynamics, a claim about this contact would further need the dynamics margin of Proposition 4(c), which we did not compute.

W2 and Q4: learning-specific benefit. We agree that the benefit is modest. The method is post-training validation, used here as a criterion for choosing a checkpoint. Where the certificate and simulated success choose differently (9 of 64 runs), the certificate picks checkpoints whose verified sets have fewer intervals (1.48 versus 1.63 on average) and a wider narrowest interval (2.09 versus 1.79 standard errors; Appendix I).

  • Protocol (Appendices E and I). The networks have one hidden layer of eight ReLU units, or sixteen in a variant, and imitate a fixed feedback law on 4,096 uniform inputs, with a separate validation set of 4,096 samples. Checkpoints are saved after 150, 300, 600 and 1,200 updates. Each run has three development and four held-out contexts (start and goal pairs), and no selector sees the held-out contexts or the calibration measurement.
  • Reliability (Section 5.3, Appendix I, Table 13; tolerance 0.05, four measurements). Certification-based selection gives 1.19 points higher held-out certificate utility than task-cost selection (95% interval [0.48, 2.09]). Of the 22 runs in which the choices differ, it is better in 19 and worse in 1, and the 56 new runs alone give 1.27 points. Its advantage over simulated-success selection, however, is only 0.38 points [0.11, 0.77]. After Holm adjustment for multiple comparisons, the gain over task cost holds at tolerances 0.025 and 0.05 but not at 0.1, and it is not significant for networks with sixteen hidden units.

Q5: multivariate costs and adaptivity.

  • Costs (Appendix J.3). With three uncertain parameters, a finite Gaussian net covering a whitened unsafe region of diameter 10 needs 4.1 × 10⁷ points to keep the total-variation penalty at 1%. Our robust scalar profile, which requires success over a box of the other parameters, takes a median of 497 ms (Table 19) but certifies only 2.3% of its domain.
  • Adaptivity (Appendix J.4). The implementation fixes the verified sets and radii before measuring (Algorithm 1), which keeps the guarantee at no cost. Naive recalibration after an abstention reaches 7.68% risk, whereas a predeclared protocol that splits δ across refinement levels is valid and costs 5.0 points. For sequential data, testing after every four measurements up to 64 (16 looks), each at level δ, has risk up to 24.1%, while spending δ/16 on each look is valid.

The revision strengthens the theoretical contribution through explicit mismatch bounds and a clearer account of when the finite-boundary calculation applies. Section 5.5 and Appendix H also extend the verifier to a planar point mass with two position degrees of freedom, four state variables, and two control inputs. The expert controller couples the axes through a shared bound on the command norm. All four walls use the same impact law and share one uncertain restitution coefficient. This adds contact sequences for the verifier to handle while leaving the calibration problem one-dimensional.

On 55 planar tasks with nonempty verified sets, the common-radius rule gains 2.30 percentage points in mean acceptance over the gap rule with four measurements, falling to 0.09 points with 256. The 64-seed study makes the checkpoint-selection gain more credible, but its advantage over simulated-success selection is only 0.38 points at tolerance 0.05.

The experimental evaluation remains entirely in simulation, including the new two-axis study. The reported gains establish a narrow statistical benefit under synthetic measurements. Whether the assumed error bounds hold for sensor readings during physical contact remains untested, limiting the practical conclusions I can draw from the revision.

I maintain my rating of 2/4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers. Two of them found the paper hard to read, so we revised it for clarity throughout. The revised PDF keeps all original section, theorem, table and figure numbers.

The paper asks one question. A verifier proves that a learned controller succeeds for every value of an uncertain physical parameter in a set S, which often consists of several separate intervals. Given a few noisy measurements of the parameter, which estimates justify claiming success, if a false claim may occur with probability at most 5%? The abstract now states this question, Section 1 works through one concrete task (new Figure 1(a)), and Section 2 states it formally.

The main changes are these.

  • The abstract is rewritten in plain words, and Sections 2 to 5 open with a sentence on what they show.
  • Each theorem is followed by a plain statement of its consequence, and the captions of Figure 1 and Tables 2 to 5 define every rule and column.
  • Section 5.5 adds a second verified system, a planar task with two coupled axes.
  • Proposition 4 and Corollary 5 (Section 4.2) handle measurement bias, an unknown noise scale, non-Gaussian errors and dynamics mismatch, each with a stated margin.
  • The checkpoint study now uses 64 training seeds instead of 8 (Section 5.3).

The main remaining limitations, stated in the paper, are that there is no hardware experiment and that the exact theory covers one uncertain parameter.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.