acceptodds
Under review as a conference paper at ICLR 2027

Why Does Agentic Safety Fail to Generalize Across Tasks?

Abstract

AI agents are increasingly deployed in multi-task settings, where the task to perform is specified at test time, and the agent must generalize to unseen tasks. A major concern in such settings is safety: often, an agent must not only execute unseen tasks, but do so while avoiding risks and handling ones that materialize. Empirical evidence suggests that even when the ability to execute generalizes to unseen tasks, the ability to do so safely frequently does not. This paper provides theory and experiments indicating that failures of agentic safety to generalize across tasks are not merely due to limitations of training methods, but can also reflect an inherent property of safety itself: the relationship between a task and its safe execution is more complex than the relationship between a task and its execution alone. Theoretically, we analyze linear-quadratic control with -robustness, and prove that in many cases, the mapping from task specification to an optimal controller has higher Lipschitz constant with safety requirements than without, yielding a Lipschitz bound of independent interest. Empirically, we demonstrate our conclusions in simulated quadcopter navigation with a neural network agent and in CRM with an LLM agent, showing that imitating safe and unsafe teachers is similarly straightforward on tasks seen in training, yet generalizing across tasks is considerably more difficult with the safe teacher. Our findings suggest that current efforts to enhance agentic safety may be insufficient, and point to a need for fundamentally different approaches.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Agentic AI, Multi-Task Control, Safety, Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mC6TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why agentic safety fails to generalize across tasks. The authors first provide theory on an analytically tractable model, namely the linear-quadratic control with H∞-robustness setting of Section 2.3. In this setting, within the mapping from the task parameter Q to the optimal controller, the Lipschitz constant under safety requirements can be arbitrarily larger than without safety requirements; the two mappings are denoted Ksafe and Kunsafe. This result is given by Theorem 1 in Section 3.3, with the global explicit upper bound on Kunsafe in Lemma 1 of Section 3.2 as an intermediate step. The authors argue from this that cross-task generalization of safety is structurally more difficult, independently of how good the training method is. Experiments validate this in three settings, namely linear-quadratic control in the infinite-sample and finite-sample regimes, neural-network quadcopter navigation with two safety variants covering banned regions and disturbance handling, and an LLM agent in a CRM benchmark, with a LLaMA-3.2 student and a GPT-5.2 teacher. Table 1 shows that imitating safe and unsafe teachers is similarly easy on training tasks, while on unseen tasks the safe teacher is clearly harder to generalize.

Critical strengths

1.A core contribution of this paper is to elevate "safety generalizes more poorly" from an empirical observation to a provable structural statement. The infinite-sample rows in Table 1, rows 1 to 4, give a separation of 1265.53 versus 14.16, with a design matching the model of Section 2.3, and rule out insufficient training as an explanation. This is a main ground for my support of acceptance. 2.Lemma 1 stands independent of the main conclusion. It gives a global explicit Lipschitz bound for the mapping from Q to the LQR, see Eq. 3, a useful complement to the classical local or implicit characterizations. Section 4.1 further shows the conclusions survive when the assumptions are violated, which is a treatment that deserves credit. 3.The theoretical part of the appendix is complete and self-contained, and it lays out the theorem's conditions in a form that can be checked item by item. The proofs supply the intermediate steps, with separate lemmas for the contraction property of the Riccati operator and for the bounds on its solution, and the background on LQR and H∞ is developed from first principles in Appendix B. This lets my concerns about the theorem's conditions be traced to specific statements rather than left as conjecture.

Critical weaknesses

1.The theorem's conclusion is narrower than the title suggests. Section 3.3 itself concedes that the separation coefficient is not always greater than one. Together with the commutativity assumption, the theory in effect covers only systems that are essentially diagonalizable. Whether the headline claim, that safety generalizes more poorly, extends to general systems therefore remains open, and this is the direct reason I cannot give Clear acceptance. 2.The central metric in the empirical part is not scale-normalized, and the coverage explanation is not ruled out. The imitation errors are not normalized by the scale of the target mapping, and safe teachers, which must avoid banned regions, naturally produce demonstrations with narrower state coverage. As a result, "safety is intrinsically harder to learn" and "data coverage is worse" cannot be told apart. The support that the last two experiments provide for the headline claim is weakened accordingly. 3.The evidence in the main text is thin. Only the linear-quadratic setting has a figure, while the quadcopter and CRM settings appear as a single table row. The paper's claim is about a phenomenon, yet readers can verify it only through one aggregate number, and learning curves or example trajectories would be more convincing.

Questions for authors

1.Can you provide a hand-checkable example, with A, B, D, R listed, for which the separation coefficient exceeds one? 2.Is there a parameter example in which the Lipschitz constant of Ksafe is smaller than that of Kunsafe? If so, where is the boundary of the separation result? 3.Safe and unsafe teachers have different demonstration distributions by construction. Can a distribution-aligned control separate mapping complexity from coverage? 4.After switching to relative error normalized by the target scale, which of the three settings still show the separation? 5.The Lipschitz result predicts that generalization difficulty grows with the distance of a task from the training set. Can the test errors in Table 1 be stratified by that distance? 6.Do both teachers use the same network capacity and hyperparameters? If the safe setting were given more capacity, would the separation shrink? 7.If the hard avoidance constraint is replaced by a safety penalty in the objective, does the separation persist? That would separate the role of hard constraints from that of safety-informed data. 8.CRM is the only modern agentic setting, yet its test gap is the smallest. Can that setting support a universal claim? The following are suggestions for future work: Extending the separation theorem to the non-commuting matrix case.

4 comments

Thank you for your detailed review and your support for acceptance, and for recognizing that our work elevates "safety generalizes more poorly" "from an empirical observation to a provable structural statement." You noted that what stands between our work and clear acceptance is whether our headline claim "extends to general systems." We have added results that directly support this (marked in blue in the revised PDF), as detailed below. We hope that, together with the strengths you describe as "a main ground for [your] support of acceptance," this allows you to recommend clear acceptance.

The theorem's conclusion is narrower than the title suggests...

Can you provide a hand-checkable example...?

Is there a parameter example in which the Lipschitz constant of Ksafe is smaller...?

First, the new Remark 1 characterizes exactly when the coefficient of Theorem 1 exceeds one. This regime is broad, e.g., including all admissible systems when AA, BB and RR are aligned and control authority is limited. Second, the new Theorem 2 shows that on the tasks diagonalized by the eigenbasis of the system, under a mild non-degeneracy condition, the Lipschitz constant is more than twice as large with safety requirements as without. We have also adjusted the phrasing of our claims accordingly.

Theorem 2 also yields a hand-checkable example. In one dimension, it gives Lip(Ksafe)≥(1+1/∣a∣)⋅Lip(Kunsafe)\mathrm{Lip}(K_{\text{safe}}) \geq (1 + 1/|a|) \cdot \mathrm{Lip}(K_{\text{unsafe}}), e.g., a ratio of 33 for a=1/2a = 1/2 and b=d=r=1b = d = r = 1. In closed form, Ksafe(q)=−min⁡{2q,1/2}K_{\text{safe}}(q) = -\min\lbrace 2q, 1/2\rbrace has slope 22, while Kunsafe(q)K_{\text{unsafe}}(q) is smooth with slope at most ∣ab∣/(r(1−a2))=2/3|ab| / (r(1 - a^2)) = 2/3. The true ratio is thus exactly 33: the bound is tight.

Remark 1 delineates the boundary of our bound, not of the separation: across all linear-quadratic configurations we evaluated, including ones violating Assumptions 1–3, we have not encountered a case where Lip(Ksafe)<Lip(Kunsafe)\mathrm{Lip}(K_{\text{safe}}) < \mathrm{Lip}(K_{\text{unsafe}}). We thus conjecture that the separation holds in general. The non-commuting case you suggest is a natural next step toward proving it, which we leave for future work.

The central metric in the empirical part is not scale-normalized, and the coverage explanation is not ruled out...

After switching to relative error..., which of the three settings still show the separation?

All errors in the continuous settings are already normalized by the target scale (Appendix I), so all three settings show the separation under this metric. Regarding coverage: in the linear-quadratic setting, both students are trained on the same tasks and states, and the separation persists. In the quadcopter setting, the fraction of states on unseen tasks with a training state nearby (Table 7) differs little between the students, and not consistently in either direction: the risk-avoidance student is best covered at small radii, yet generalizes far worse.

The evidence in the main text is thin...

We have added example quadcopter trajectories of both students on seen and unseen tasks (Figure 8).

Can the test errors in Table 1 be stratified by that distance?

Our Lipschitz argument bounds the error on an unseen task by its distance from the training tasks, scaled by the Lipschitz constant. Both students are trained and tested on the same tasks, so these distances are identical, and the gap stems from the Lipschitz constant. We therefore test this factor directly: across random linear-quadratic systems, the difference between the two students' generalization gaps (unseen minus seen tasks) grows with the ratio of Lipschitz constants (Figure 2).

Do both teachers use the same network capacity and hyperparameters? If the safe setting were given more capacity, would the separation shrink?

In the linear-quadratic setting, the teachers are the exact optimal controllers; in the quadcopter setting, they share an architecture and reach comparable performance; and in CRM, both are the same LLM, differing only in the prompt. Moreover, all data we use comes from tasks the teachers perform well on, so targets are (near-)optimal throughout. Teacher capacity is thus not a factor, and both students share the same architecture and hyperparameters.

If the hard avoidance constraint is replaced by a safety penalty...?

The risk-avoidance teacher is in fact trained with a soft penalty for entering banned regions rather than a hard constraint (Appendix I.4), and the risk-handling teacher has no banned regions. Both exhibit the separation.

...Can that setting support a universal claim?

Our claim is qualitative rather than universal: safety requirements make generalization across tasks harder, even when imitation on seen tasks is equally easy. CRM exhibits this: the safe student's error grows by 57% from seen to unseen tasks, compared to 7% for the unsafe student. CRM errors are also bounded, so gaps are naturally smaller in scale than in the other settings.

Dear Reviewer mC6T,

Thank you again for your review. As the discussion period ends on November 18, we wanted to kindly check whether our response has addressed your concerns. If so, we would greatly appreciate it if you would consider raising your score. We would be happy to clarify anything further.

Best wishes,

Authors

Thank you for the detailed response. On the experiments, the control in the linear-quadratic setting is clean and the coverage question is handled there. The empirical part is still thin, in that each setting is reported at a single operating point, so the evidence shows that the two targets differ in difficulty rather than that the safety requirement accounts for the difference. I find the experimental side not yet complete, and I am keeping my rating at 2.

Thank you for the follow-up. As you note, the linear-quadratic setting is cleanly controlled, and our other settings follow the same design: both students face the same systems and tasks with the same learner, and imitate (near-)optimal teachers that differ only in whether they ensure safety. It is therefore the safety requirement that accounts for the difference in difficulty between the two targets.

Thank you again for the constructive discussion.

Reviewer_v45cOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This article theoretically studies learning difficulty under safety constraints via the proxy setting of linear-quadratic control. The paper also provides an empirical investigation of practical settings where the theory does not apply. The core claim is that generalization under safety constraints is more challenging due to the Lipschitz constant of the optimal controller under safety constraints being larger than the optimal controller without safety constraints.

Critical strengths

While at times lengthy (e.g. lengthy definition of a standard linear control problem) the writing was easy to follow. The topic is timely and relevant. In principle, the combination of theory and empirical evaluation provides a strong foundation for an analysis of safety generalisation issues.

I did not check the proofs in detail, but at a first glance, they appear to be in good shape.

Critical weaknesses

The core critical weakness of this article is that Theorem 1 does not show what is otherwise discussed in the article. As acknowledged in line 300, Theorem 1 does not, in fact, show that the Lipschitz constant under safety constraints is larger than the Lipschitz constant without safety constraints. Instead, the Lipschitz constant under safety can also be smaller than the Lipschitz constant without. This breaks all conclusions taken from this result in the paper, as they rely on the Lipschitz constant under safety being larger.

What the paper instead shows is that there are some settings in which the Lipschitz constant under safety can be significantly larger. This result is insufficient to support the conclusions drawn for generalisation.

This critical issue is tied in with another weakness of the paper. While it makes sense to me to study linear-quadratic control as a simplified setting, it is not clear that this setting bears any resemblance to realistic agentic tasks. It, therefore, remains questionable if the results can be transferred. This is further exacerbated by the focus on risk avoidance and risk handling as strategies for safety. Due to the enormous diversity of safety issues in agents, I am not convinced these settings are sufficiently representative. Furthermore, due to the issue with representing the theory described above, the overall conclusion of the paper does not even hold in this restrictive setting

Questions for authors

The issues underlying my review are too severe to be addressed in the rebuttal.

5 comments

Thank you for your review, and for recognizing that our writing is "easy to follow," that our work provides "a strong foundation for an analysis of safety generalisation issues," and that our proofs "appear to be in good shape." We believe your main concern stems from a misunderstanding, which we clarify below, along with your remaining concerns. Your feedback has also led to several improvements to the paper, marked in blue in the revised PDF. If you find our responses satisfactory, we would greatly appreciate it if you would consider raising your score. Should any questions remain, we would be glad to discuss them.

...Theorem 1 does not, in fact, show that the Lipschitz constant under safety constraints is larger... the Lipschitz constant under safety can also be smaller...

...the overall conclusion of the paper does not even hold in this restrictive setting

We fear there may be a misunderstanding. Theorem 1 is a lower bound Lip(Ksafe)≥c⋅Lip(Kunsafe)\mathrm{Lip}(K_{\text{safe}}) \geq c \cdot \mathrm{Lip}(K_{\text{unsafe}}). When c<1c < 1, it does not certify the separation, but it does not imply the reverse either. Line 300 states only that cc can be below one, and the paper never claims or observes a reversal. On the contrary, Lip(Ksafe)\mathrm{Lip}(K_{\text{safe}}) is larger in every linear-quadratic configuration we evaluated, including where c<1c < 1 and where Assumptions 1–3 are violated.

Following your feedback, we have also made two additions to the theory:

  • Explicit condition for separation (new Remark 1). We characterize exactly when Theorem 1 certifies Lip(Ksafe)≥Lip(Kunsafe)\mathrm{Lip}(K_{\text{safe}}) \geq \mathrm{Lip}(K_{\text{unsafe}}), and show that this holds for a broad range of systems, e.g., for all admissible systems when AA, BB and RR are aligned and control authority is limited.

  • Uniform separation on restricted tasks (new Theorem 2). On the tasks diagonalized by the eigenbasis of the system, we prove that, under a mild non-degeneracy condition, the Lipschitz constant is more than twice as large with safety requirements as without.

These additions show the separation is broad. We therefore conjecture that the separation holds in general, and proving this is an important direction for future work.

...This result is insufficient to support the conclusions drawn for generalisation.

The Lipschitz constant is an established surrogate for the difficulty of generalization [2, 3, 8, 9]. Empirically, in all of our settings, imitating the safe and unsafe teachers on training tasks is similarly easy, while generalizing to unseen tasks is considerably harder with the safe teacher. We have also added experiments linking the two directly: across random linear-quadratic systems, the difference between the two students' generalization gaps (unseen minus seen tasks) grows with the ratio of Lipschitz constants (Figure 2), and the safe quadcopter teachers have 2.5–4.6 times larger Lipschitz constants (Table 4).

...it is not clear that this setting bears any resemblance to realistic agentic tasks...

Our approach of pairing theory in a simplified setting with broader experiments, and drawing takeaways that extend beyond the formally analyzed setting, is common in modern machine learning research [1, 4, 5, 6, 7, 10, 11]. In line with this, our experiments go beyond the theory, to quadcopter navigation with neural network agents and an LLM agent in CRM, where the separation persists.

...I am not convinced these settings are sufficiently representative.

The two notions follow the two main formulations of safety in reinforcement learning and control, covered at length in Appendix A: risk avoidance corresponds to constraint satisfaction (e.g., constrained MDPs), and risk handling to robustness to model mismatch and adversarial disturbances (e.g., H∞H_\infty control and robust MDPs). In LLM agents, they correspond to prohibited actions and to appropriate responses to risky interactions, respectively, and our CRM experiment covers both. We do not claim the two notions are exhaustive.

—

References

[1] Arora et al. "On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization"

[2] Bartlett et al. "Spectrally-Normalized Margin Bounds for Neural Networks"

[3] Golowich et al. "Size-Independent Sample Complexity of Neural Networks"

[4] Gunasekar et al. "Implicit Regularization in Matrix Factorization"

[5] Lee et al. "Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent"

[6] Saxe et al. "Exact Solutions to the Nonlinear Dynamics of Learning in Deep Linear Neural Networks"

[7] Schulman et al. "Trust Region Policy Optimization"

[8] Sokolić et al. "Robust Large Margin Deep Neural Networks"

[9] Stone. "Optimal Global Rates of Convergence for Nonparametric Regression"

[10] Tsipras et al. "Robustness May Be at Odds with Accuracy"

[11] Xie et al. "An Explanation of In-context Learning as Implicit Bayesian Inference"

Reviewer_v45cOpenReview ↗

Requests for Additional Clarification

Dear authors, Thank you for your response. While misunderstandings can occur, your response rather strengthens the point I raised in my review. As you state in your response, safety forcing a larger Lipschitz constant in linear-quadratic control is currently merely a conjecture. There are only proofs for certain settings. I am happy to reconsider the submission if you can convince me of the relevance of these settings. I will state concrete questions below to which a convincing answer would change my evaluation.

our work provides "a strong foundation for an analysis of safety generalisation issues,"

First of all, this sentence in your rebuttal misrepresents my review. I wrote, "In principle, the combination of theory and empirical evaluation ..." before this part, which does not refer to your submitted work but to the general methodology. I also found that other parts of your rebuttal do not address the actual points raised in my review. I will clarify them below to avoid any misunderstandings.

...This result is insufficient to support the conclusions drawn for generalisation.

The Lipschitz constant is an established surrogate for the difficulty of generalization [2, 3, 8, 9]. Empirically, in all of our settings, imitating the safe and unsafe teachers on training tasks is similarly easy, while generalizing to unseen tasks is considerably harder with the safe teacher. We have also added experiments linking the two directly: across random linear-quadratic systems, the difference between the two students' generalization gaps (unseen minus seen tasks) grows with the ratio of Lipschitz constants (Figure 2), and the safe quadcopter teachers have 2.5–4.6 times larger Lipschitz constants (Table 4).

To clarify, I agree that the Lipschitz constant is a suitable proxy for the difficulty of learning. My issue is that the submission does not contain a general proof of the discussed relationship between safety and generalization. Without generality, I do not see strong relevance for agent safety.

  • Q1: Why are the settings for which the submission proves the relationship between safety and LQC particularly relevant for agent safety?

...it is not clear that this setting bears any resemblance to realistic agentic tasks..

Our approach of pairing theory in a simplified setting with broader experiments, and drawing takeaways that extend beyond the formally analyzed setting, is common in modern machine learning research [1, 4, 5, 6, 7, 10, 11]. In line with this, our experiments go beyond the theory, to quadcopter navigation with neural network agents and an LLM agent in CRM, where the separation persists.

I fully agree that such a theoretical analysis can be insightful, as I already stated in my review. However, for it to be useful, one has to argue why the setting that is theoretically analysed is a suitable proxy for the practical setting. I am missing this link between LQC and realistic agent tasks.

  • Q2: Why is LQC a good proxy for agent tasks?

Thank you for the clarification and for the concrete questions. As the sentence appeared under strengths, we read it as referring to our work, and we apologize for the misreading. We are glad we agree that the Lipschitz constant is a suitable proxy for the difficulty of learning, and that pairing theory with experiments can be insightful; below we address the remaining point, the link between our theory and agent safety. We hope that our answers will lead you to reconsider your evaluation.

Q2: Why is LQC a good proxy for agent tasks?

LQC retains the features central to our question: an agent acts repeatedly in an environment and its actions shape future states, the policy depends on the task, and safety requirements (risk handling via H∞H_\infty-robustness) have a canonical formulation. Indeed, LQC is a special case of the general formulation of learning a multi-task agent with safety requirements (Section 2.2), which also covers the LLM agent in our CRM experiment (Section 4.3), so our theory and experiments study the same object: the mapping from task to policy. Moreover, theoretical analysis of LQC is a well-established approach for gaining insight into reinforcement learning and the control of nonlinear systems [5, 8, 9, 11, 12].

More broadly, in modern learning theory, it is standard to study complex systems through simplified models that retain their relevant structure, e.g., multi-index models for feature learning in neural networks [1, 3], and linear attention and linear SSMs for sequence models [2, 4, 10, 13]. As in these works, our experiments go beyond the theory, and its conclusions persist: with both neural network (quadcopter) and LLM (CRM) agents, imitating the safe and unsafe teachers is similarly easy on training tasks, but generalizing to unseen tasks is considerably harder with the safe teacher, and the safe quadcopter teachers have 2.5–4.6 times larger Lipschitz constants (Table 4).

Q1: Why are the settings … relevant for agent safety?

Theorem 2, added following your review, specializes our analysis to systems that decompose into independent modes (Assumption 2), a structure arising in practice, e.g., in systems of identical interconnected subsystems [6, 7]. It shows that in such cases the separation holds uniformly: safety requirements more than double the Lipschitz constant over all tasks aligned with these modes.

More broadly, the paper claims that safety requirements can make generalization across tasks fundamentally more difficult, i.e., that this difficulty can stem from safety itself rather than from limitations of training methods. This existence claim is established by proofs in explicit and broad settings and does not require a general proof; the conjecture in our previous response concerns all instances of LQC, which goes beyond what the paper claims. Empirically, the separation far exceeds the bound of Theorem 1 (Figures 1 and 3).

Finally, the relevance of all of our proven settings to agent safety is that they isolate the cause: since the optimal policies are characterized exactly and safety is learned perfectly on training tasks, the learner is ruled out, an isolation that experiments with practical agents cannot provide; our experiments complement this by showing that the phenomenon extends to such agents. We will add this discussion to the paper.

—

References

[1] Abbe et al. "The Merged-Staircase Property: A Necessary and Nearly Sufficient Condition for SGD Learning of Sparse Functions on Two-Layer Neural Networks"

[2] Cohen-Karlik et al. "Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets"

[3] Damian et al. "Neural Networks Can Learn Representations with Gradient Descent"

[4] Emami et al. "Implicit Bias of Linear RNNs"

[5] Fujinami et al. "Domain Randomization is Sample Efficient for Linear Quadratic Control"

[6] Hovd and Skogestad. "Control of Symmetrically Interconnected Plants"

[7] Massioni and Verhaegen. "Distributed Control for Identical Dynamically Coupled Systems: A Decomposition Approach"

[8] Razin et al. "Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial States"

[9] Toso et al. "Meta-Learning Linear Quadratic Regulators: A Policy Gradient MAML Approach for Model-Free LQR"

[10] von Oswald et al. "Transformers Learn In-Context by Gradient Descent"

[11] Zhang et al. "Multi-Task Imitation Learning for Linear Dynamical Systems"

[12] Zhang et al. "On the Stability and Convergence of Robust Adversarial Reinforcement Learning: A Case Study on Linear Quadratic Systems"

[13] Zhang et al. "Trained Transformers Learn Linear Models In-Context"

Reviewer_5ZVmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks into the reasons for the difficulties of safe actions learning and generalization to unseen tasks. The author highlights that the lipschitz constant would be an important factor and uses the theory from linear quadratic control to prove that under some conditions, the lipschitz constant of the map for the safe actions is bigger than the unsafe one which makes it difficult to learn. Experiments on linear quadratic control, quadcopter and llm agent show the difficulties of generalization to safety.

Critical strengths
  1. The question is important and interesting. It inspires researchers to think about the safety of the learning, why it is difficult to learn and possible research directions to solve the problems.
  2. The theory looks good and sound to prove the relationship between the lipschitz constant of two settings.
  3. The experiments include different settings, such as classical control problem and llm agents.
Critical weaknesses
  1. The paper only uses Lipschitz constant as the metric to measure how difficult the map is to generalize across tasks with safe constraints and the connection is mostly discussed by citing prior works and listing as a limitation. To my understanding, the claim of the related paper does not mean that only a larger lipschitz constant leads to the difficulties of learning and generalization There are still a lot of factors. It is not clear that how the author handle with this issue both in theory and experiments.
  2. Even if the lipschitz constant is the key factor, the claim that safe actions have bigger lipschitz constants is proved under specific assumptions such as assumption 1 to 3. The author also mentioned that "Our experiments go beyond the theoretical limitations above." which the reviewer is happy to see while this could be caused by other factors since the conditions do not hold in other experimental settings.
  3. The derivation of Lipschitz constant is based on linear quadratic control while the input doesn't take safety specification and related observations into consideration to my understanding. In the experiments, like quadcopter, the information of obstacles is not encoded into the observations. The reviewer thinks that the safety itself introduces an additional layer of information that needs to be learned. However, I doubt that whether this addition should be attributed to the Lipschitz constant.
Questions for authors
  1. Could you give more detailed information about the Lipschitz constant and the other factors that you can control in the experiments that related to the generalization? When we change other factors, does the generalization follow the the analysis in this paper?
  2. The dynamics in the llm agent experiments are hard to be written into an exact formulation. I think it would be better to clarify this point using an estimated Lipschitz for this task and check the numerical connections.
  3. Is the information provided to the student already enough for the learning to generalize to unseen tasks? If the constraint-related observations are given as input, does the analysis still hold?
7 comments

Thank you for your review, and for recognizing that the question we study "is important and interesting," and that our theory "looks good and sound". Your feedback has led to several improvements to the paper, marked in blue in the revised PDF. If you find our responses satisfactory, we would greatly appreciate it if you would consider raising your score.

The paper only uses Lipschitz constant as the metric...

...When we change other factors, does the generalization follow the the analysis in this paper?

The Lipschitz constant is an established surrogate for the difficulty of generalization [1, 2, 3, 4], which is why our theory focuses on it. In our experiments, we control the other factors that could affect generalization: the safe and unsafe students are trained and evaluated on the same tasks and splits, with identical architectures and hyperparameters, and the teachers are the exact optimal controllers (linear-quadratic), share an architecture (quadcopter), or are the same LLM (CRM). Both students perform comparably on tasks seen in training, so the main difference between them is the mapping they imitate. Following your feedback, we have added experiments directly linking the Lipschitz constant to generalization. Across random linear-quadratic systems, the difference between the generalization gaps (unseen minus seen tasks) of the safe and unsafe students is positive throughout, and generally grows with the ratio of Lipschitz constants (Figure 2). In the quadcopter setting, the Lipschitz constants of the safe teachers are 2.5 (risk avoidance) and 4.6 (risk handling) times larger, in line with their students' generalization gaps (Table 4). We have also added a coverage check: in the quadcopter setting, the fraction of states on unseen tasks with a training state nearby differs little between the students (Table 7).

...the claim that safe actions have bigger lipschitz constants is proved under specific assumptions such as assumption 1 to 3... this could be caused by other factors since the conditions do not hold in other experimental settings.

Following your feedback, we have strengthened the theory in two ways. First, the new Remark 1 characterizes exactly when the coefficient of Theorem 1 exceeds one. This regime is broad, e.g., it includes all admissible systems when AA, BB and RR are aligned and control authority is limited. Second, the new Theorem 2 shows that on the tasks diagonalized by the eigenbasis of the system, under a mild non-degeneracy condition, the Lipschitz constant is more than twice as large with safety requirements as without. Beyond the theory, the systems in our linear-quadratic experiments are drawn at random and do not satisfy Assumption 2, yet the separation persists, both in Lipschitz constants and in generalization.

...In the experiments, like quadcopter, the information of obstacles is not encoded into the observations... the safety itself introduces an additional layer of information that needs to be learned. However, I doubt that whether this addition should be attributed to the Lipschitz constant.

Is the information provided to the student already enough...? If the constraint-related observations are given as input, does the analysis still hold?

We agree that safety adds a layer that must be learned; this is in fact our thesis, with the Lipschitz constant quantifying the added complexity it induces in the relationship between a task and its safe execution. Our focus is on imitation learning, where the student has no explicit access to the safety risks, and must learn how to handle them from the teacher's demonstrations alone. This reflects practical settings in which safe behavior is conveyed through demonstrations rather than specified explicitly, and it is precisely the difficulty of learning safety this way that we study. Concretely, the safe and unsafe students receive the same inputs, and differ only in the demonstrations they imitate; in the quadcopter setting, these inputs (state and target position) are exactly those from which the teachers act. Since the banned regions are fixed across tasks, adding them as input would only append a constant, leaving the mapping from tasks to actions, and hence our analysis, unchanged.

...I think it would be better to clarify this point using an estimated Lipschitz for this task and check the numerical connections.

In CRM, actions are discrete text strings, so the Lipschitz constant is not well defined. We therefore estimate it in the continuous settings, where the new experiments above connect it numerically to generalization.

—

References

[1] Bartlett et al. "Spectrally-Normalized Margin Bounds for Neural Networks"

[2] Golowich et al. "Size-Independent Sample Complexity of Neural Networks"

[3] Sokolić et al. "Robust Large Margin Deep Neural Networks"

[4] Stone. "Optimal Global Rates of Convergence for Nonparametric Regression"

Thank you for the detailed response and the clarification. While it helps me understand some parts of the paper, I still have some concerns:

To my understanding of the references you shared, [1]-[3] not only discuss the Lipschitz constant but also other factors like the margin. In [4], they also consider the the smoothness order in the rate, which are also important for generalization. Could you clarify how you control the related factors? I think safety constraints may also affect the smoothness order of the mapping.

For safety, we consider a control barrier function controller (equation CBF-QP in Sec. II-C [1]). If the constraint is affine and does not depend on Q, this mapping is nonexpansive and its Lipschitz constant w.r.t. Q is no larger than the unsafe one. I think this is a very common setting, and it contradicts the claim in the paper.

These make me still confused.

[1] Ames, Aaron D., et al. "Control barrier functions: Theory and applications." 2019 18th European control conference (ECC). Ieee, 2019.

Thank you for engaging with our response. We are glad it helped clarify parts of the paper, and address your two remaining concerns below: other factors affecting generalization, and safety filters such as CBF-QP. We will add both discussions to the paper. If you find our responses satisfactory, we would greatly appreciate it if you would consider raising your score.

...[1]-[3] not only discuss the Lipschitz constant but also other factors like the margin. In [4], they also consider the the smoothness order in the rate... Could you clarify how you control the related factors? I think safety constraints may also affect the smoothness order of the mapping.

We agree that the Lipschitz constant is not the only factor affecting generalization. As you note, [1, 2, 3] tie it to generalization alongside other factors; among these, margin arises only in classification, whereas the mappings we analyze have continuous outputs (controllers and thrusts).

We also agree that safety can affect the smoothness order, and in our setting it lowers it: the unsafe mapping is smooth in the task, whereas the safe mapping can have kinks (e.g., already in the scalar case; see the closed form in our response to Reviewer mC6T). By [4], a lower smoothness order means slower rates, so this reinforces, rather than confounds, the separation we prove. For a given order, the minimax error grows with the corresponding smoothness constant [5], which for the first order is precisely the Lipschitz constant our theory bounds.

Empirically, we measure generalization error directly, which reflects all such factors jointly. The safe and unsafe students share architecture, training procedure, tasks and inputs, perform comparably on training tasks, and differ only in the teacher they imitate. The difference in generalization is thus due to the mapping being learned, whichever properties of the mapping (Lipschitz constant, smoothness order, or others) contribute to it.

For safety, we consider a control barrier function controller... this mapping is nonexpansive and its Lipschitz constant w.r.t. Q is no larger than the unsafe one... it contradicts the claim in the paper.

CBF-based safety filters are indeed common, and we agree with your calculation. CBF-QP is a form of risk avoidance (keeping the state out of unsafe regions), which our theory does not cover (Section 5) and which we study empirically. In fact, your example helps clarify where the added complexity comes from.

A CBF-QP is a filter applied on top of the unsafe controller: it minimally corrects each action to satisfy the constraint, so when the constraint does not depend on the task, safety is decoupled from the task by construction. Such corrections can come at the expense of the task, e.g., QP-based CBF controllers can get stuck at undesirable equilibria [6]. The filtered policy is thus safe but generally suboptimal for the task. In our experiments, the safe teacher instead optimizes the task objective under the safety requirement, so how it stays safe depends on the task, e.g., the route around the banned regions depends on the target. Notably, in this risk avoidance setting, the banned regions are fixed across tasks as in your example, yet the safe teacher has a 2.5 times larger Lipschitz constant (Table 4), and its student generalizes considerably worse (Table 1).

Moreover, a CBF requires an explicit barrier function certifying safety, which is often unavailable for agents (e.g., in our CRM setting), where safe behavior is instead conveyed through demonstrations (Section 1). Where a task-independent safety filter is available, it can avoid the difficulty, possibly at the expense of the task, in line with the direction suggested in our conclusion: simplifying the relationship between tasks and safety characteristics.

Overall, your example sharpens our claim: the added complexity arises when safe behavior must adapt to the task, as with teachers that optimize under safety requirements, and not when safety is enforced by a task-independent filter. The paper claims that safety can make the mapping more complex, not that it always does, and the discussion we will add makes this explicit. We hope this resolves the remaining confusion, and would be happy to clarify further.

—

References

[1] Bartlett et al. "Spectrally-Normalized Margin Bounds for Neural Networks"

[2] Golowich et al. "Size-Independent Sample Complexity of Neural Networks"

[3] Sokolić et al. "Robust Large Margin Deep Neural Networks"

[4] Stone. "Optimal Global Rates of Convergence for Nonparametric Regression"

[5] Györfi et al. "A Distribution-Free Theory of Nonparametric Regression"

[6] Reis et al. "Control Barrier Function-Based Quadratic Programs Introduce Undesirable Asymptotically Stable Equilibria"

Comments on the paper

1 comment

Dear reviewers and AC,

We thank the reviewers for their thoughtful feedback. We have uploaded a new version of the paper that includes all changes we committed to (marked blue in the PDF). Below is a list of the main changes.

Theory

  1. Explicit condition for separation (Remark 1). We characterize exactly when Theorem 1 certifies Lip(Ksafe)≥Lip(Kunsafe)\mathrm{Lip}(K_{\text{safe}}) \geq \mathrm{Lip}(K_{\text{unsafe}}), and show that this holds for a broad range of systems, e.g., for all admissible systems when AA, BB and RR are aligned and control authority is limited.

  2. Uniform separation on restricted tasks (Theorem 2). On the tasks diagonalized by the eigenbasis of the system, we prove that, under a mild non-degeneracy condition, the Lipschitz constant is more than twice as large with safety requirements as without.

Experiments

  1. Lipschitz constant vs. generalization (Figures 2, 6 and 7). Across random linear-quadratic systems, in both the infinite- and finite-sample regimes, the difference between the generalization gaps (error on unseen tasks minus error on seen tasks) with and without safety requirements is positive throughout, and generally grows with the ratio between the respective Lipschitz constants.

  2. Lipschitz constants of the quadcopter teachers (Table 4). The Lipschitz constants of the safe teachers are 2.5 (risk avoidance) and 4.6 (risk handling) times larger than that of the unsafe teacher, in line with the generalization gaps of their students.

  3. Coverage check (Table 7). In the quadcopter setting, the fraction of states on unseen tasks that have a training state nearby differs little between the students, and not consistently in either direction, so differences in coverage do not explain the separation.

  4. Example quadcopter trajectories (Figure 8). Trajectories of the safe and unsafe students on tasks seen and unseen in training illustrate the separation.

Presentation

  1. Refined phrasing (abstract, introduction, Section 3). We slightly refined several statements to more precisely reflect the scope of our theory.

We believe these changes have substantially strengthened the paper, and thank the reviewers for the feedback that led to them.

Best wishes,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.