acceptodds
Withdrawn from ICLR 2027

Safe Probing Certifies What a Robot Can Do, Not What It Cannot

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Safe physical probing can establish that a robot meets a task demand, while a safety envelope can prevent it from ruling out feasibility. We formalize this asymmetry using a capability signature , the supremum level an embodiment reaches along a task's binding capability axis in a scene, compared with a demand computed from scene geometry alone. Probes are restricted to , whereas is defined over with . Within the stated response class, feasibility can be certified with finitely many probes whenever the true response at some safe grid amplitude exceeds , without a regularity constant. In the scalar-response observation model, for demands below the trivial upper bound, infeasibility has no uniformly valid test with non-trivial power: an infeasible response admits an indistinguishable extension beyond the envelope that changes the verdict. The upper identification endpoint remains at its trivial bound, even under smoothness or Lipschitzness with an unknown constant. SOMA-Probe combines a variance-adaptive, anytime-valid lower bound with four structural witnesses, each based on a stated, falsifiable, dimensionless order property. It selects probe amplitudes by risk-penalised information gain and returns Certified-Feasible, Certified-Infeasible, or Uncertified. A companion result bounds coverage for probing policies using the prescribed certification rules. The benchmark comprises four manipulation tasks and sixty embodiment perturbations, eighteen outside the parameter vector exposed to the identification baselines. On the blind perturbations, certification coverage is , compared with for the strongest safe-exploration baseline. The reported mis-certification rate is at a nominal level of , excluding on D5a and the optional regularity branch. In a tier analysis across all sixty perturbations, the end-to-end advantage over ASID is concentrated in the eighteen perturbations outside its fitted parameter vector.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot capability certification, safe active probing, embodiment uncertainty, partial identification, confidence sequences, abstention

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fK7ROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Robots must be able to assess whether the task they are supposed to execute is feasible. This judgment may change over time, as the robot wears down, links might bend, or parts might be replaced. The authors propose a method to classify tasks as feasible, infeasible, or uncertifiable. The last category may, for instance, be necessary if execution is hindered by a safety envelope.

Critical strengths
  1. The problem itself seems important. The authors consider an important problem setting, which could make this a valuable contribution.

  2. The authors compare their method to ten baseline methods, thus providing substantive evidence.

Critical weaknesses
  1. Overall, I found the paper quite difficult to follow. The paper mentions many concepts but doesn't properly introduce some of them. Each concept has its own notation, which the reader must recall throughout the paper. For instance, it's unclear what the "witnesses" are or what "explicit shape assumptions" they add. The constant forward-referencing is more confusing than helpful. I didn't find a clear problem formulation in the corresponding section. At the high level, it is clear what the paper wants to do, but the details remain somewhat vague. Overall, this makes the paper difficult to access and limits its potential.

  2. A proper literature review is missing. The connections to SafeOpt are also unclear to me; SafeOpt seems to address a very different problem setting. This makes it hard to position the paper and evaluate its relevance.

  3. The assumptions underlying the theoretical statements are not clear. What does it mean for observations to be bounded? To my knowledge, the empirical Bernstein mixture supermartingale used requires several assumptions. For example, bounded random variables, a martingale difference sequence, finite variance, mixing conditions, etc. I didn't see those assumptions explicitly verified. The authors didn't make any explicit assumptions about the noise level, other than that it has zero expected value. This makes it difficult to verify the theoretical results.

Questions for authors
  1. Could you clarify what assumptions you make about the system noise?

  2. Can you position your paper with respect to the literature?

  3. What exactly is the problem setting? Trying to classify feasible tasks from limited probing? Which would force us to also extrapolate beyond what has been tried?

Reviewer_xrLFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an interesting problem: what a robot can reliably determine about its own capabilities through safe physical probing. The authors introduce SOMA-Probe, which combines confidence bounds, structural witnesses, and adaptive probing to provide three possible decisions: feasible, infeasible, or uncertified. The paper also provides theoretical analysis and extensive simulation experiments under different embodiment perturbations.

Critical strengths

I appreciate the effort put into the theoretical formulation and experimental evaluation. The distinction between certifying feasibility and infeasibility is interesting, and the results suggest that capability-based probing can be useful when conventional parameter identification fails to represent certain physical changes.

Critical weaknesses

My main concern is the validity of the structural assumptions used for infeasibility certification. The paper correctly explains that infeasibility cannot generally be certified from observations restricted to the safe region. However, the proposed witnesses rely on additional assumptions about the response outside the observed range. These assumptions may be difficult to verify in practice, especially when the robot has experienced unknown physical degradation. The D5a experiments illustrate this issue, where the W4 certification error reaches 10.4%. Although the authors discuss this limitation, I would like to understand how the method could determine whether a particular witness is reliable before using it to certify infeasibility.

I am also concerned about how safety is guaranteed during probing. The safety envelope is constructed using the nominal robot model, which may no longer accurately describe the robot after degradation. The experiments report a safety-abort rate of 6.8% at the main operating point, increasing substantially with more permissive probing. This raises a question about what safe probing means in practice. The authors should clarify whether these safety violations represent actual unsafe interactions or conservative monitoring triggers, and how the approach could maintain safety when the underlying model is unreliable.

The current formulation depends on several strong assumptions about the task and environment. In particular, the binding capability axis is assumed to be known, and task demand is computed accurately from the scene. The experiments also use a welded anchor, virtual wall, and state restoration to isolate probing from task execution. While these choices make the theoretical analysis more manageable, they may be difficult to reproduce on physical robots. I would appreciate more discussion or experiments showing how sensitive the certification results are to errors in identifying the capability axis, estimating task demand, or isolating probing from the task.

The comparison with parameter identification methods deserves further consideration. The paper shows that most of the improvement over ASID comes from D5 perturbations, which are deliberately outside the parameter vector used by that baseline. This is a reasonable setting for studying model mismatch, but it also makes the advantage over parameter identification somewhat expected. The ablation using the same probing data is helpful, although I would still like to see comparisons with more flexible identification or adaptation methods that can represent these changes. This would make the contribution of capability-based certification more convincing.

The empirical evaluation is extensive, but its generality remains uncertain. All experiments are conducted in MuJoCo using one robot platform and four tasks with predefined capability axes. In addition, the benchmark applies retention criteria based on oracle agreement and nominal-policy performance. It would be helpful to clarify how these criteria affect the final evaluation population and whether the improvements remain similar without this filtering. Testing on a different robot or a wider range of physical interactions would also strengthen the paper.

Finally, I think the practical advantage of the three-way decision framework needs further evaluation. The comparison merges uncertified cases into certified infeasibility for the two-way baseline, which naturally leads to more abandoned tasks. However, other decision policies could handle uncertainty differently, for example by selecting a conservative action or using a risk-sensitive execution threshold. Comparing against these alternatives would help establish the benefit of the proposed decision framework beyond the particular two-way baseline considered.

Questions for authors
  1. How can the robot determine whether the structural assumptions required by W1–W4 are satisfied for an unknown embodiment? In particular, how would the framework prevent incorrect infeasibility certificates in cases similar to D5a, where W4 exceeds the target error rate?

  2. The safety envelope is determined using the nominal model, but the robot may have undergone significant physical degradation. How can the proposed framework guarantee safe probing under such model mismatch? What do the reported safety-abort events represent physically?

  3. How sensitive is the method to errors in the predefined capability axis and scene-derived demand? Have the authors evaluated cases where the task bottleneck is not known accurately or changes during interaction?

  4. Since the main improvement over ASID is concentrated in perturbations outside its fitted parameter vector, would a more flexible identification model reduce this performance gap? What advantage would capability-based certification retain under that comparison?

  5. Could the authors clarify how the benchmark retention criteria were applied and whether the results remain consistent across all initially generated task and perturbation configurations?

  6. Have the authors considered comparing the three-way decision framework with alternative uncertainty-aware execution policies, rather than only a two-way policy that treats all uncertified cases as infeasible?

Reviewer_6ttpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper considers the problem of probing-based robot learning under safety constraints and engages with the question of what can be certified, and how, from a finite number of experiments ("probe primitives") carried out by the robot under a pre-established safety envelope. The approach aims to construct a ternary certificate probe that outputs either certified-feasible, certified-infeasible or uncertified, and presents some analysis on the probability of incorrect certification. The experimental results use a simulated robotic manipulator arm over 4 different tasks, introducing a large number of possible changes to the robot's physical parameters ("degradation" conditions). The distinction between uncertified and certified-infeasible helps inform the decision on whether the robot should attempt a task. The paper concludes positive and negative feasibility evidence is rendered asymmetric by the safety envelope.

Critical strengths

The paper investigates an important open problem area in robot learning—behavior certification in the presence of safety constraints—and is doing so by bridging promising ideas from statistical certification and safe exploration.

The experimental results contribute an extensive evaluation benchmark that introduces a varied array of physical degradation conditions on a common type of robot manipulator arm platform, opening the door to possible extension to physical robotics experiments.

Critical weaknesses

The exposition is out of order. The writing uses technical terminology from the onset without previously introducing it, which makes the first few pages nearly impossible to parse without first going through later sections of the paper.

  • The "binding physical axis" is mentioned multiple times in the introductory sections without definition, clarification, or examples; "safe observations" are placed at the center of the opening narrative, but the reader is not told what a safe observation actually is; the same then happens with the "task demand" and the "safety envelope".
  • The formal statement of contributions alone is entirely presented in terms of unexplained concepts: a "safe-grid margin", the "amplitude domain", the "trivial bound", a "certificate family".
  • The Problem Formulation and SOMA-Probe sections then use mathematical notation that is either never defined or only explained in later paragraphs or sections (e.g., q˙\dot{q}, vtolv_{\text{tol}}, ff).
  • Even in later sections, multiple concepts are not clarified explicitly: what is the safety "abort" mechanism, how and when are the safety criteria "tripped", etc.

Overall, I found the exposition genuinely difficult to follow (it took me a long time to work my way through the paper, and I repeatedly found myself having to jump back to reinterpret previous passages once the necessary concepts were finally introduced, out of order), and I think the paper needs a deep reorganization to reach a publication-ready state.

The paper's contributions are not framed in the relevant literature. The discussion of related work in the main paper is very sparse—the core is in the Introduction, which only cites 11 prior publications (with a strong skew towards very recent work).

  • While a key motivation are the limitations of "body identification" in robotics, the discussion misses the large literature in system identification and fault detection and isolation, which has extensively studied runtime physical degradation in robotics and autonomous systems (including cases in which the type of degradation is unmodeled or, as the paper puts it, "outside the fitted parameter vector"). Relevant examples range from classical Kalman and particle filters for online parameter tracking and anomaly detection to more recent efforts in robot learning, such as the work by Antoine Cully, Angela Schoellig, Andreas Krause, Andrea Bajcsy, and their groups.
  • The paper has no main-text references (and only a few in the appendix) to the safe robot learning literature, which is central to the proposed motivation and formulation. This absence unfortunately seems tied to the overall disconnect between the paper's abstraction of "safety envelopes" and the manner in which most modern safety filters modulate robot learning and exploration (see the point below).

The technical relevance to safe robot learning is unclear. The proposed framework applies only to quasi-static robot "probe primitives", varied along a discrete "amplitude" 1-D grid; in this setting, the safety envelope is modeled as an amplitude cap hard-coded at various seemingly hand-designed "levels" so as to limit the achievable "capability signature" at several tasks.

  • The immediate issue is that this a significant over-simplification of actual safety-filtered robot learning: safety filters are fundamentally dynamic (rather than quasi-static), with multidimensional (rather than scalar) safety envelopes, and thanks to this they are able to keep robots from executing dangerous transient maneuvers, but not from reaching otherwise feasible steady-state configurations.
  • The deeper issue is that this impacts the key insight of the paper, as summarized in the title: that safe probing does not certify what a robot cannot do; that is, that learning under a safety-enforcing mechanism prevents the robot from ascertaining, let alone achieving, the full realization of its capability at a given task. This central claim is incorrect. It relies on the idea that using any form of safety filter necessarily over-constrains the robot's behavior, barring it from achieving its full performance at a task. In fact, this premise has been disproven both theoretically and empirically in recent years: modern studies on robot learning under safety filtering (notably by Bejerano, Brunke, and Schoellig at ICRA 2025, and by Oh, Nguyen, Hu, and Fisac at IASEAI 2026) have found that state-of-the-art safety filters do not fundamentally limit achievable task performance, and often boost learning. (In a nutshell, the reason is that a good safety filter only bans actions that result in catastrophic failures, so, in the paper's terminology, there are no "feasible extensions beyond the envelope".)

Given the above discrepancies, it is unfortunately not clear that the proposed framework would be applicable, or necessary, in realistic robot learning settings. And in any case, it is clear that the paper's title and "bigger picture" takeaway, as stated, are incorrect.

Questions for authors
  1. How do you determine the mapping from ζ\zeta to the threshold values in Eq. (1), in general and specifically in your experimental setting? Is there a principled interpretation of ζ\zeta, or is the mapping arbitrary? What is the "nominal model" you use to compute these margins, just a deterministic, zero-degradation robot simulator or something more sophisticated like a stochastic model?

  2. Why do you need to introduce online safety overrides ("aborts") separate from the safety envelope? Is this due to a discrepancy between the "nominal" model and the deployment condition, or is the issue that the quasi-static assumption breaks online even without degradation?

  3. Are the online safety overrides intervening before a safety violation takes place or only once the violation has already happened? It's not fully clear, but it appears like it's the latter—but, if so, what is the point of aborting after you have already crashed?

  4. How does your framework extend to non-scalar settings? Can it work for general constrained action/policy spaces?

  5. How does the validity of the Certified-Feasible guarantee relate to the validity of A_safe actually being safe? Can "surprise online aborts" break the probability bounds on mis-certification, or are they already accounted for?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.