acceptodds
Under review as a conference paper at ICLR 2027

Responsiveness Verification: Will Predictions Change? How Much? How Often?

Abstract

Machine learning models are often used in applications where their inputs change due to routine interactions, strategic manipulation, or noise. In such settings, models can undermine safety as these changes can lead them to predict over regions of the input space they have not seen. We propose to address these challenges by measuring *responsiveness*—the probability that a model output attains a target prediction under feasible interactions defined by the interaction model. We develop algorithms to estimate responsiveness for any machine learning model, and a framework to specify broad classes of interaction models. We pair these algorithms with statistical guarantees that support practical validation. We demonstrate how our tools can promote safety and reliability across domains by detecting preclusion in recidivism prediction, estimating the cost of gaming in content moderation, and testing the robustness of benchmarks for LLMs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
responsiveness, safety, verification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_7mAoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how predictions change under feasible input interactions. It defines responsiveness as the probability of attaining a target output, estimated through constrained sampling and assessed using binomial statistical tests. The authors propose a unified auditing framework and demonstrate applications to recourse, gaming, and LLM safety.

Critical strengths
  1. This paper proposes a useful common interface for specifying actionable changes, joint constraints, and stochastic downstream effects.
  2. The evaluation is under black-box setting and the reuse of sampled inputs across models is a practical design.
Critical weaknesses
  1. There seems to be a limited distinction from the closest prior work:"Feature Responsiveness Scores: Model-Agnostic Explanations for Recourse ICLR 2025", already introduces responsiveness probabilities, rejection sampling for continuous features with downstream effects. The statistical guarantees largely apply standard binomial inference rather than establish a substantial new theoretical result.
  2. The guarantees do not support several safety interpretations. Low responsiveness under uniformly sampled actions does not establish that a prediction is fixed or that recourse is impossible. Similarly, low random-action gaming probability does not establish robustness against a strategic attacker. The conclusions should distinguish distribution-dependent probabilities from existential or worst-case guarantees.
  3. Sampling correctness is insufficiently established (in Alg.1). The statistical results require i.i.d. samples, but set union discards duplicates in discrete spaces and can prevent termination when fewer than (n) distinct points are reachable. The algorithm does not justify the claimed generality or the assumptions underlying certification.
  4. The main algorithmic advantage is not demonstrated. No runtime, acceptance-rate, scaling, or partitioning ablation establishes efficient sampling under complex constraints. Fig. 2 evaluates statistical accuracy versus sample count, rather than the computational cost of generating samples. This leaves the framework’s principal computational contribution insufficiently supported.
Questions for authors
  1. Does retraining without age or drug-abuse features reduce preclusion as claimed in Sec. 4.2? Allowing these features to change during an audit does not establish the effect of removing them from model training.
3 comments

Thank you for your time and feedback! We appreciate your comments and concerns, which have helped us to refine our paper. We would like to address them below:

There seems to be a limited distinction from the closest prior work [...]

We would like to highlight two main differences:

Purpose: Cheon et al. [1] propose Feature Responsiveness Scores as an explanation method for algorithmic recourse. We propose an auditing framework for safety with statistical guarantees. These two methods are typically useful at different stages of the model lifecycle. Practitioners would use a post-hoc explanation method like Feature Responsiveness Scores after they deploy a predictive model (e.g., to provide explanations that are compliant with regulations). In contrast, one would use our testing framework before deployment to ensure that their models assign safe predictions under interactions.

Scope of interactions (1-D vs. n-D): Feature Responsiveness Scores aim to measure the responsiveness of each feature; it only considers changes in a single feature (see Definition 3 in [1]). We work with a superset of interactions considered in [1] --- feasible interactions across all features.

Feature Responsiveness Scores asks the question:

How responsive is feature jj?

In contrast, we ask:

How responsive is f(x)f(x)?

All features of xx may have responsiveness scores of 0 while f(x)f(x) still has positive responsiveness. This happens when no single-feature action attains the target, but at least one multi-feature action does.

Therefore, feature responsiveness scores in [1] are a special case/application of our definition of responsiveness, where we only consider single-feature interactions. Hence, our sampling algorithm and the optimization problem associated with the algorithm is more complex than the ones in [1]. Because we are dealing with a much larger space of interactions, the partitioning step (lines 1–3 in Algorithm 1) is necessary for the sampling to be computationally tractable.

We have clarified this in lines 100-110 of the related work section in our revision (new text in bold, citations in here correspond to references in the paper):

“On the technical front, our proposal is most closely related to research on algorithmic recourse. Existing methods find recourse actions [see e.g., 24, 39, 53], often for specific model classes [39, 50], or certify the feasibility of recourse only over discrete feature spaces [25]. We generalize the notion of responsiveness proposed by Cheon et al. [7], who introduce Feature Responsiveness Scores to explain opportunities for recourse through changes to individual features. Our framework instead tests responsiveness under feasible interactions across all features. To verify responsiveness, we build on the model-agnostic machinery of Kothari et al. [25], who construct the set of all points reachable through interactions. We extend their approach to be compatible with any dataset and replace enumeration with sampling, which makes verification computationally viable for complex interaction models with probabilistic downstream effects. As a result, our approach can certify the feasibility of recourse over any dataset, requires only black-box access, and returns practical guarantees.”

statistical guarantees largely apply standard binomial inference rather than establish a substantial new theoretical result.

Section 3 indeed uses classical statistical results. This is by design, as our contribution is not a new statistical test, but rather a new object to test. Responsiveness unifies a broad class of safety failures as a single testable quantity. The technical work is what makes standard inference possible here:

  1. A formal interaction model (Definition 1) that captures realistic interactions through actionability constraints and downstream effects.
  2. A sampler for reachable sets (Algorithm 1) that handles complex interaction spaces that standard samplers do not support.
  3. Sensitivity and ablation procedures (Section 3.2) that let practitioners stress-test their own assumptions and trace failures to specific constraints.

This lets us rely on standard statistical inference, which gives practitioners exact guarantees that they are already familiar with.

The guarantees do not support several safety interpretations [...] conclusions should distinguish distribution-dependent probabilities from existential or worst-case guarantees.

Our guarantees are exact for existential claims (some feasible action attains the target). A single sampled action that is responsive proves that recourse exists, or that a prediction can be gamed.

Universal claims (no feasible action attains the target) are different. With only black-box query access, no finite procedure can certify this, since the region that attains the target prediction can be arbitrarily small. Instead, we provide a statistical certificate with a clear interpretation: when we reject the null hypothesis (i.e., ρ(x)≥τ\rho(x) \geq \tau), with confidence 1−α1-\alpha, fewer than τ\tau of feasible actions attain the target. Practitioners choose τ\tau and α\alpha to match the stakes of their application, and our sample-size results (Proposition 3, Remark 4) tell them how many samples they need.

We default to uniform sampling because it does not require a cost model nor an attacker model, which we rarely know. When practitioners need stronger claims, our framework offers two options:

  1. Enumeration (Section 4.1) gives exact responsiveness when the action set is discrete.
  2. Importance sampling (Section 3.3) concentrates samples on actions a strategic attacker is likely to take.

Sampling correctness is insufficiently established (in Alg.1)

Thank you for catching this. Our implementation retains duplicate samples, but the set union notation in Algorithm 1 did not reflect that. We have revised Algorithm 1 to be precise (most notably, it now returns an indexed sequence (xi′)i=1n(x'_i)_{i=1}^n). We have also clarified in Section 2 that we repeat the sampling procedure independently. In Appendix C.1, we now state the assumptions required for the algorithm to terminate almost surely and return i.i.d. samples from the specified distribution.

The main algorithmic advantage is not demonstrated

With the extra space, we added an additional experiment in Section 4.1 that evaluates runtime and the benefit of partitioning when generating reachable points. In short, our sampling method with partitioning reduces the total runtime by at least 66% and 34% over naive sampling and enumeration, respectively. Thus, our sampling procedure can make verification practical when constructing the full reachable set is too costly. We can also apply our sampling procedure to continuous or mixed (discrete and continuous) settings, which enumeration cannot handle.

Does retraining without age or drug-abuse features reduce preclusion as claimed in Sec. 4.2?

This is a great question and yes, it does reduce preclusion! When we train without the two features, only 2.7% of defendants predicted to recidivate remain effectively fixed. We have added this detail at the end of Section 4.2 in our revision.


[1] Cheon, Seung Hyun, et al. "Feature responsiveness scores: Model-agnostic explanations for recourse." International Conference on Learning Representations 2025.

Reviewer_7mAoOpenReview ↗

Response to authors

Thanks for the detailed answers to my concerns and questions. I believe much of my concerns have been well responded and addressed. I will raise my rating from 2 to 3.

Reviewer_YkfMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a framework for auditing how model predictions respond to feasible input changes, measuring responsiveness as the probability of reaching a target output under a specified interaction model. It combines actionability constraints and stochastic downstream effects with partitioned sampling and statistical tests to estimate responsiveness and assess it against chosen thresholds. Experiments on recidivism prediction, bot detection, and LLM safety demonstrate how the framework can identify difficult-to-change predictions, susceptibility to manipulation, and sensitivity to prompt variations.

Critical strengths

The paper addresses a well-motivated gap in model evaluation: predictive accuracy alone does not establish whether a model responds appropriately to feasible input changes. The demonstrations across recidivism prediction, bot detection, and LLM safety show why this matters and support the practical value of a general auditing framework.

Critical weaknesses

The novelty relative to Feature Responsiveness Scores is insufficiently clarified. That work already defines responsiveness probabilistically and estimates it through sampling under actionability constraints, including downstream effects. The present paper supports broader interactions and statistical testing, but does not clearly separate inherited components from new contributions or isolate the benefits of these extensions.

Questions for authors
  • What are the precise methodological advances over Feature Responsiveness Scores, particularly in the responsiveness definition, interaction model, and sampling procedure?
  • Can you demonstrate a setting that the prior framework cannot adequately handle, and quantify the benefit of the proposed extensions—for example, the efficiency of partitioned sampling under joint constraints and stochastic downstream effects?
2 comments

Thank you for your time and feedback! We appreciate your comments and concerns, which have helped us to refine our paper. We would like to address them below:

The novelty relative to Feature Responsiveness Scores is insufficiently clarified

What are the precise methodological advances over Feature Responsiveness Scores, particularly in the responsiveness definition, interaction model, and sampling procedure?

We would like to highlight two main differences:

Purpose: Cheon et al. [1] propose Feature Responsiveness Scores as an explanation method for algorithmic recourse. We propose an auditing framework for safety with statistical guarantees. These two methods are typically useful at different stages of the model lifecycle. Practitioners would use a post-hoc explanation method like Feature Responsiveness Scores after they deploy a predictive model (e.g., to provide explanations that are compliant with regulations). In contrast, one would use our testing framework before deployment to ensure that their models assign safe predictions under interactions.

Scope of interactions (1-D vs. n-D): Feature Responsiveness Scores aim to measure the responsiveness of each feature; it only considers changes in a single feature (see Definition 3 in [1]). We work with a superset of interactions considered in [1] -- feasible interactions across all features.

Feature Responsiveness Scores asks the question:

How responsive is feature jj?

In contrast, we ask:

How responsive is f(x)f(x)?

All features of xx may have responsiveness scores of 0 while f(x)f(x) still has positive responsiveness. This would happen when no single-feature action attains the target, but at least one multi-feature action does.

Therefore, feature responsiveness scores in [1] are a special case/application of our definition of responsiveness, where we only consider single-feature interactions. As a result, our sampling algorithm and the optimization problem associated with the algorithm is more complex than the ones in [1]. Because we are dealing with a much larger space of interactions, the partitioning step (lines 1–3 in Algorithm 1) is necessary for the sampling to be computationally tractable.

We have clarified this in lines 100-110 of the related work section in our revision (new text in bold, citations here correspond to references in the paper):

“On the technical front, our proposal is most closely related to research on algorithmic recourse. Existing methods find recourse actions [see e.g., 24, 39, 53], often for specific model classes [39, 50], or certify the feasibility of recourse only over discrete feature spaces [25]. We generalize the notion of responsiveness proposed by Cheon et al. [7], who introduce Feature Responsiveness Scores to explain opportunities for recourse through changes to individual features. Our framework instead tests responsiveness under feasible interactions across all features. To verify responsiveness, we build on the model-agnostic machinery of Kothari et al. [25], who construct the set of all points reachable through interactions. We extend their approach to be compatible with any dataset and replace enumeration with sampling, which makes verification computationally viable for complex interaction models with probabilistic downstream effects. As a result, our approach can certify the feasibility of recourse over any dataset, requires only black-box access, and returns practical guarantees.”

Can you demonstrate a setting that the prior framework cannot adequately handle, and quantify the benefit of the proposed extensions—for example, the efficiency of partitioned sampling under joint constraints and stochastic downstream effects?

Absolutely! For instance, Kothari et al. [2] proposed what is equivalent to falsification in our Table 1, and introduced tools for falsification in discrete domains. However, their work is restricted to discrete settings, which limits where practitioners can use their method in practice. Furthermore, our work focuses on tasks beyond exact falsification in estimation and testing responsiveness.

Moreover, even in discrete settings, sampling can often be the better choice. We have added a runtime comparison between sampling and enumeration in Section 4.1 of our revision. We found that sampling reduces the time it takes to generate reachable points by at least 34% (47 minutes vs. 71 minutes) compared to enumeration. Hence, sampling can make verification practical when constructing the full reachable set is too costly.


[1] Cheon, Seung Hyun, et al. "Feature responsiveness scores: Model-agnostic explanations for recourse." International Conference on Learning Representations 2025.

[2] Kothari, Avni, et al. “Prediction without preclusion: Recourse verification with reachable sets.” International Conference on Learning Representations 2024.

Thank you for the clarification and the revisions. After careful consideration, I decide to keep my score.

Reviewer_zjgyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces the framework of responsiveness, a test-based method to evaluate how model outputs and safety may change under different interaction conditions. It offers a sampling-based approach to evaluation and validates that it produces results on recidivism algorithms, liver transplant matching, LLM benchmarking, and bot detection.

Critical strengths

The proposed method is flexible and applicable in several different domains to audit model safety and highlight possible interaction concerns prior to deployment.

The paper is technically sound and offers substantial implementation details.

Critical weaknesses

Several of the chosen examples have a known interaction safety violation. While this method successfully recovers them, it does so post-hoc, with interaction models designed with the knowledge of what errors may occur. A more thorough test would be to apply the method to a scenario where the error case is not known, and design interaction models for that case or explain why it is out of scope.

The paper does not sufficiently differentiate this method from existing error analyses. For example, in the LLM benchmarking scenario, translation and fictional roleplaying are known jailbreak techniques. The paper would be strengthened with an explanation of what additional information the responsiveness adds on top of a traditional jailbreaking assessment.

Questions for authors

What evaluation method was used for LLM responses on sorry bench?

4 comments

Thank you for your time and feedback! We appreciate your comments and concerns, which have helped us to refine our paper. We would like to address them below:

A more thorough test would be to apply the method to a scenario where the error case is not known, and design interaction models for that case or explain why it is out of scope

You are correct in observing that the example use cases in Section 4.2-4.4 have “known safety violations.” This is because when we are testing for safety, we often have specific mechanisms that we want to be robust against (e.g., we don’t want LLM benchmarks to be susceptible to rephrasing that doesn’t alter semantic meaning). However, our framework doesn't require this prior knowledge. When we don't have a specific mechanism in mind, we can construct a "general" interaction model based on the features themselves. This way, the interaction model captures all realistic changes to the input. In fact, this is the case in Section 4.1, where we work with the german credit dataset. Here, we test whether predictions are responsive to change (i.e., have recourse) under a general interaction model. If a prediction is not responsive, we conclude that the individual may not have recourse under all realistic/feasible interactions, not with respect to a particular subclass of interactions.

The paper would be strengthened with an explanation of what additional information the responsiveness adds on top of a traditional jailbreaking assessment.

One of the goals of our case studies is to show the generality of our framework, in particular, that it can subsume existing practices such as recourse verification and robustness evaluations of language models. Specifically in the LLM case study, we cast the problem of evaluating model robustness to jailbreaking strategies as an instance of a responsiveness verification. This allows practitioners to make use of our statistical framework to select the sample size, compare models, or perform analyses such as those described in Sec. 3.3.

What evaluation method was used for LLM responses on sorry bench?

As detailed in Appendix H (Appendix I in the revision), we use the official fine-tuned judged model according to the sorrybench prompting protocol.

We just wanted to follow up and see if you’ve had a chance to read our comments! With only a couple more days left in the discussion period, please let us know if you have additional questions or concerns. If our response has resolved your concerns, please consider raising your score!

Thank you for your detailed response. I remain concerned that the method requires a specific designed known safety violation, as Section 4.1 does not include details of a discovered safety violation, only verification of the estimation error and test reliability. I would ask either for additional details on that experiment, or a further scoping in the paper of what scenarios this method works on. I maintain my initial score.

Thank you for your reply!

There seems to be a slight misunderstanding here! As we have stated in our response, our method does not require "a specific designed known safety violation." Having said that, we did agree with the reviewer that most of our empirical demonstrations were in applications where we did consider a specific set of interactions for safety.

For example, in our experiments on recidivism prediction (Section 4.2), we measure the responsiveness of predictions when we "clear" criminal history features. This is indeed a specific set of interactions we had in mind. This is based on how the problem was motivated: the risk score model developed in Pennsylvania assigned fixed predictions to males under 21, regardless of their criminal history.

On the other hand, in Section 4.1, we measure the responsiveness of predictions under a general interaction model. This interaction model would simply describe what "realistic" actions individuals can take (Appendix D has the exact interaction model). Here, a safety violation occurs when predictions have low (or zero) responsiveness, meaning no action attains the target, so the prediction may be fixed. This doesn't require knowledge of exactly how predictions may be fixed. We have made additional revisions to Section 4.1 to make this clearer. In lines 342-347:

Here, we work with a general interaction model comprised of constraints that are inherent to each feature's semantic meaning. We set Yˉx={1}\bar{Y}_{x} = \{1\} (i.e., "good" credit) and consider the subset of individuals who are predicted to be a customer with "bad" credit by the model (f(x)=0f(x) = 0). In this case, low responsiveness would suggest that the prediction may be fixed since there aren't many actions (if at all) that the individual can take to be considered a "good" applicant, potentially barring access from credit.

We have also added the following detail in Section 3.2, where we mention how interaction models determine responsiveness (lines 264-268):

In general, practitioners construct interaction models based on inherent constraints on the inputs to specify what actions are viable in real life. In practice, safety applications often have specific failure mechanisms that we want to be robust to. In this case, practitioners would craft a targeted interaction model to measure responsiveness under specific interactions of their interest. In either case, practitioners must decide which features are actionable, what constraints govern joint changes, and whether there are downstream effects.

We also plan on further revising Section 4.1 to make it more cohesive to demonstrate both our statistical guarantees and safety violations under a general interaction model.

One thing we do need in our framework is the target outcome. This should be clear in every safety application by asking "what do we consider a safety violation?" But, we don't need to know how exactly those safety violations occur in order to design the interaction model.

Hopefully, this clears things up!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.