acceptodds
Under review as a conference paper at ICLR 2027

AI Assurance with Double Machine Learning

Abstract

AI assurance is the problem of testing whether artificial intelligence (AI) and machine learning (ML) systems adhere to a set of policies. Machine learning systems are particularly hard to assure since they are error-prone, unfair and biased. For example, is an auditor's claim that somebody is manipulating the system actually true or is there a more mundane explanation owing to an inaccurate or unfair classifier? To this end, we advocate the use of double machine learning (DML) as a framework for testing falsifiable assurance claims against ML systems. The approach directs ML upon itself to explain away its own workaday behaviors — behaviors that otherwise interfere with behavioral ML assurance tests. We study DML on the problem of detecting manipulation in content moderation systems. We find that DML allows us to isolate the effects of manipulation even in the presence of other interfering phenomena. On manipulation detection, DML achieves an AUC of 0.92 compared to fairness and statistical baselines which achieve AUCs around 0.76. We find DML succesfully detects a variety of manipulation attempts, including loss function manipulation, intentional bias amplification and the direct manipulation of its moderation decisions. Strikingly, in the latter case, DML estimates directly reflects the percent of manipulated content moderation decisions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI assurance, double machine learning, content moderation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Yn9mOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper is an application of double machine learning (DML) [Chernozhukov et al., 2024] to AI Assurance. AI Assurance is the process of auditing a AI to test its compliance with a set of policies. More precisely, the article deals with content moderation with simulations using Jigsaw dataset.

The paper contains interesting data transform to augment the Jigsaw dataset with synthetic binary topic variable, to test in particular how DML can be used to detect content moderation manipulation under varying levels of confounding selection rate ratio skew.

Among other results, the authors show that even using a part of the available features, DML can improve the precision of the estimation.

They also claim that DML can be applied for other wilder scenarios (no doubt about that).

Overall Assessment

This paper presents an elegant idea on an important problem, and I would be glad to raise my score towards acceptance if my concerns are addressed. Reproducibility (W2) is the decisive point for me: without the code and further experimental details, I would have to lower my recommendation. The related work (W1) could be improved, but I do not consider it a deal-breaker.

Critical strengths

S1. Clarity and presentation. I would like to thank the authors for the quality and clarity of their writing. Reading this paper is a real pleasure. The introduction sets up the problem very effectively, and the overall flow is excellent: whenever a question came to mind, the following paragraph answered it.

S2. Timeliness and originality. The research is timely. While surrogate models have been used for AI assurance before, this is the first time I have seen the problem approached through the lens of Double Machine Learning (DML). The idea is simple, clever, and appears to work well, which makes it a valuable contribution to the field.

Critical weaknesses

W1. Related work and choice of baselines. The related work section would benefit from a broader coverage of the (black-box) auditing literature, in particular recent work such as:

[1] Robust ML Auditing Using Prior Knowledge (ICML 2025). This paper proposes a general black-box auditing framework and highlights the importance of prior knowledge. It would be valuable to discuss how the role and accuracy of such prior knowledge relate to the subsets of feature dimensions (25%, 50%, 75%, 100%) considered in the submitted paper. I am not asking for an experimental comparison, but rather for a discussion that positions this work within the existing literature.

[2] Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation (AIES 2018). This is one of the earlier works using a surrogate model to audit a model. Although its setting differs, it should be acknowledged in Section 2 (again, as a discussion rather than an experimental comparison).

[3] The DSA Transparency Database: Auditing Self-Reported Moderation Actions by Social Media (Proc. ACM HCI, 2025). This work seems particularly relevant and should be discussed (see Q2).

In addition, the baselines are not described, and the paper does not explain why they are relevant for the comparison (see Q4). Finally, I would encourage the authors to connect the detection of manipulation under confounding skew to the literature on fairwashing, which addresses a closely related concern.

W2. Reproducibility. My main concern is the reproducibility of the work. I acknowledge the reproducibility statement at the end of the paper, but in its current form it is insufficient. Since DML is an existing method, a central contribution of the paper lies in its application to AI assurance and in the accompanying experimental evaluation. In that context, sharing the code is essential for the community to verify and build upon the results. Moreover, the data augmentation procedure described in Sections 4.1 and 4.2 should be explained and illustrated in more detail, for example with pseudo-code, a worked example, the intuition behind each step, key takeaways, and plots of the resulting data distributions. Such material is typically provided in the appendix alongside the code.

Questions for authors

Clarification requests

  • Q1. Is the DML coefficient reported in Section 4 the average treatment effect discussed in Section 3?

  • Q2. What are the links and implications of your work with respect to [3]? For instance, could DML be used to verify the information reported in the DSA Transparency Database? Conversely, could the information in that database be used to improve the DML-based approach?

  • Q3. Could you clarify the role of Kapusta’s framework in your approach?

  • Q4. Could you explain how the baselines were chosen? A short description of each baseline and the reasons for selecting them would help. This also relates to the reproducibility of the comparison (W2).

  • Q5. Could the data transformations described in l. 212–215 and l. 258–261 be used in other contexts beyond this paper? Do you consider them a contribution of the paper? Is there a more standard approach in the literature? Please provide more details, including at least one illustrative example and the code.

Suggestions for improvement

  • If space allows, please elaborate on the statement in l.159–161. This point is very important, as it reflects a major limitation shared by much of the black-box auditing literature.

  • Related to Q4 and Q5: please provide additional information to improve reproducibility, including examples illustrating the data transformations. It would also be very helpful to report the raw values of the metrics before thresholding (rather than only summaries such as “4/5 fails”), and to release the code.

  • Please consider adding a short conclusion or a list of key takeaways at the end of the paper.

Minor remark

Please use a consistent format for percentages and proportions throughout the paper (e.g., “1” vs. “100%”, “94.7” vs. “0.947”).

2 comments
AuthorsOpenReview ↗

Updated paper for better reproducibility and attached code for data generation

Thank you very much for your suggestions and comments. We have updated the paper in response, focusing specifically on reproducibility and related work. We also attached code for the data generation process. Please see details below.

W2. Reproducibility.

*We updated the paper to lay out the data generation process in Appendix A, lines 756--881. We include (A.1) an algorithmic description, (A.2) a pedagogical example of a toy dataset that walks through each step, along with visuals to help support understanding, and (A.3) a brief derivation of the calculation involved to achieve the toxicity rate ratio. We also attached the code for a python implementation as supplemental material.

*We updated the paper with descriptions of the baselines including our motivation for including them. See Section 4.2 lines 252—264.

*We attached code for data generation, but releasing the code repository is difficult since it's treated as highly restricted in our organization. We are working on a way to get it released. In the mean time we updated the paper with your reproducibility suggestions. As an anecdote, much of our code and findings have been reproduced internally on private data. We are confident in the reproducibility of our results and will try to get the code released to make it easier for others to reproduce and build upon it. *We added Appendix C to describe the content moderation classifiers and hyperparams more clearly at the request of another reviewer.

W1. Related work.

Thanks for these suggestions on related work. We agree with all them. We updated the related work (Section 2) to describe the relation to black box auditing and designed the discussion in part around your questions about how ideas from specific papers might interact with our work (Lines 99-116). We also updated the fairness paragraph to include fairwashing (Lines 130-136).

Clarification requests:

Q1. Correct. We updated the paper changing the first few occurrences of coefficient to "DML ATE estimate" and use "DML estimate" in all the proceeding ones for consistency with rest of paper.

Q2. This is a great question. We updated the related work to include a discussion about our work and DSA. We believe that DSA can help us form hypotheses, understand the data better in order to come up with more comprehensive nuisance features, and also help do root cause analysis. See Lines 107--116 in Section 2 for more detail.

Q3. More specifically their idea of an assurance claim aligns with ours as a way to turn a nebulous AI assurance problem into a falsifiable scientific hypothesis for which we can gather evidence. However, our view of AI assurance is not quite as broad as theirs, as we consider mitigation a separate but related problem.

Q4. Yes --- we updated the paper with a new section describing and motivating the baselines (See Section 4.2 lines 252—264).

Q5. We updated the paper with an appendix that includes the description of the data generating process and an illustrative example. We hadn't thought about it as a main contribution of the paper, but perhaps an ancillary one to help answer our research questions. But perhaps it could be used in other contexts. Any time you want to generate a binary variable that is not completely disconnected from the text, drawing a random hyperplanes seems like a reasonable thing to do, and certainly better than flipping a coin. There's possibly an interesting connection to simhash here, in that enough random topics might eventually uncover the cosine similarity between embeddings of content pairs (i.e, number of topics in common means higher cosine similarity). This gives some weight to the idea that our topics are semantically grounded in the text even if they are not directly interpretable. As for skew, we reassign points closest to the hyper plane to achieve skew but over/under sampling is another thing that is done for example, in the distribution-shift literature to rebalance data.

Suggestions for improvement:

  • "If space allows..." There's a lot to say here but based on real world experience we often have a strong prior about how a black box ML model should work for a particular task and what features it ought to use. We can also use whatever resources, if any, we have available (e.g., people, documentation, policy documents) to peer a little deeper into the box. From here, we can test how faithful our nuisance models are to the black box model on the data and iterate as needed.

  • "Related to Q4 and Q5...." We updated the paper to address both Q4 and Q5.

  • We updated the paper with a conclusion.

  • Regarding percent/proportions we made several consistency edits and will continue to go through the paper.

I thank the authors for engaging the discussion. As the discussion is ending really soon, I'm not able to read again the paper in the mean time or dig for additional answers.

I'll take everything into account during the internal discussion to give my final score.

Reviewer_JyByOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the use of Double Machine Learning (DML) for detecting manipulation in AI content moderation systems. The authors propose using DML to estimate the effect of topic membership on moderation decisions while accounting for confounding content features. Experiments on the Jigsaw dataset with synthetically manipulated classifiers show that DML reduces false positives and achieves better manipulation detection performance than statistical baselines.

Critical strengths
  1. The paper addresses an important problem in AI assurance, providing a meaningful motivation for distinguishing manipulation from normal variations in model behavior.

  2. The experiments cover different manipulation settings and show promising results, with DML consistently outperforming the statistical baselines.

Critical weaknesses
  1. It is unclear how DML distinguishes intentional manipulation from unintended model bias. Both can produce similar treatment effects, so detecting a topic-dependent effect does not necessarily indicate intentional manipulation.

  2. The method assumes that the auditor's observed features capture all relevant confounders, which is difficult to guarantee in black-box auditing. Randomly dropping embedding dimensions does not adequately test the effect of missing important confounders. Moreover, DML may fail to detect manipulation that operates through features included as control variables.

  3. The baselines do not explicitly account for confounding, making the comparison less convincing. Stronger baselines, such as covariate-adjusted regression or propensity score matching, are needed to establish the advantages of DML.

  4. The paper contains several inconsistencies in its reported experimental results. For example, the 25%-feature setting reports an AUC of 0.86 and an outcome-model F1 of 0.643 in the text, while Table 2 reports 0.883 and 0.697, respectively. Figure 3 also reports inconsistent F1 scores and decision thresholds. These discrepancies should be clarified to ensure the reliability of the experimental results.

Questions for authors
  1. How can DML distinguish intentional manipulation from unintended model bias when both may produce similar treatment effects?

  2. Can the authors evaluate the robustness of DML when important confounders are missing, or when manipulation operates through features used as control variables?

  3. Can the authors compare DML with stronger baselines that account for confounding?

1 comment
AuthorsOpenReview ↗

Updated paper with suggested control variable experiment

Thanks so much for the reviews. We updated our paper in response to your comments and criticisms and gently push back against others. More detail follows.

Critical weaknesses

  1. Yes, sorry, this is a bit confusing. In general determining "intentional" is impossible as that may be known only to the one committing the "manipulation." In our specific case we intended the manipulation in some cases by manipulating the topic and loss function in certain ways whereas in other cases, we intended no manipulation but it happened anyway due to random chance or bias in the model. So we had ground-truth intention but this is only a specific case that would not generalize to all cases. As for distinguishing between them, the intuition here is that if there is any bias in the content moderation model, the DML outcome model will have a similar bias and explain it away. Whereas if the content moderation ML model is directly manipulated, the DML outcome model will lack the more blatant modeling change and so even if it has the capacity to learn the manipulation, it will have trouble generalizing this manipulation to held out data, which is a key part of DML. Please note that this is just an intuition and not a hard fact or theorem that we prove.

  2. We gently push back against the limitation of missing confounders as a critical weakness. In the black box setting we consider (without the ability to entertain counterfactuals) it is hard to do much better. In the worst case, we miss all confounders, then we're no worse off than the baselines of correlation. However such a case is rare as in practice we have strong priors about how the ML models involved in content moderation work and what features they use, especially on familiar modalities like text, images, audio and video. Moreover, we can use outcome model fidelity to the black box model as a proxy for how well we've captured the confounders. If fidelity is bad then we know we need to keep improving the model. Finally, there is some work on handling missing unknown confounders in the literature which we cite in the paper, but save for future work.

As for features that operate as control variables, we agree that this could be a problem. In the extreme, it would likely break DML. We updated the Appendix with an experiment that explores this case. Here we include the topic features as nuisance features so the manipulation is on topics and the topics are determined by the topic dimensions, which are included in our control features. Moreover, in our second set of experiments we included loss function manipulation which biases the content-moderation features to learn topic-manipulations. This is a less severe case of what you suggest but we still find that DML works well.

  1. It is an interesting idea to consider other causal inference frameworks but it is not clear what the advantage of including them would be. We argue in the paper for the need to handle content moderation features as confounders for manipulation detection. We then take a well established causal inference method, DML, and show that it makes massive improvements over methods that do not handle the features. So for this reason it is unclear what the point of including other causal frameworks would be. Moreover, applying them to the problem is a full-blown research endeavor.

  2. Thanks for letting us know! We have gone over the paper again carefully and fixed them in the updated version. We will continue to fix any issues as we see them.

Questions for the author

  1. Please see the response to critical weakness 1.
  2. We updated the paper wth Appendix A2 to include a much more severe form of the condition in which manipulation occurs through the control variables. The main paper does include both of these evaluations already and finds DML works in each case. Reasonable people could argue that missing 75% of the embedding dimensions counts as "missing important confounders" and this is one of the experimental conditions we include in the paper.
  3. Please see response to critical weakness 3.
Reviewer_AepAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the umbrella problem of AI assurance, under which the authors singularly focus on the problem of testing falsifiable assurance claims, particularly attribution of poor performance (along some axis), in AI systems. Here, the paper proposes use of the existing framework of double/debiased machine learning (DML) for the stated problem. Empirical results presented concentrate on manipulation detection in content moderation tasks; in this setting, the paper demonstrates that DML is highly discriminative in its ability to attribute performance quirks to malicious system manipulation vs. noise from the classifier, and in fact, detect manipulation even when noisy classification is present. Furthermore, DML exceeds existing baselines for detection in terms of AUC performance and can detect manipulation stemming from a diverse set of interference pathways.

Critical strengths
  • For the setting of manipulation detection in content moderation, the empirical analyses are extremely thorough / comprehensive and examine success/failure modes in detail across metrics of interest and demonstrate utility of DML in this context
  • The presentation of the problem addressed, methodology used and tasks examined is very clear
  • Limitations of the analysis are clearly stated
Critical weaknesses
  • An examination of existing methodology in a new application is certainly useful but the application here is very narrow, much more narrow than the title suggests as it focuses exclusively on manipulation detection in content moderation, solely on the Jigsaw dataset and no other AI assurance setting
  • Very few baselines compared, only the 4/5ths rule and fairness, would be useful to have more comparison to other causal methods to understand where DML specifically receives the boost in performance, and some methods from hypothesis testing (classical statistics), and the AI governance/audit and “washing” literature such as fairwashing, privacy washing, etc.
  • No ablative studies for how the content moderation classifiers are trained, which might impact the differential results, or alternatively, there are no citations that bolster the procedure used to obtain the content moderation classifiers (i.e. that this is what is generally used in practice or considered in previous literature on content moderation)

Overall, I felt the paper feels a bit more suited for a workshop at this stage, but certainly could be improved by either broadening the scope of the AI assurance tasks considered, or doing a deeper dive into the utility of DML in the context of content moderation via incorporation of more baselines and review of when IRM/DML fails.

Questions for authors
  • On the topic of baselines as stated in the weaknesses, an additional point here would be to understand or discuss if DML can be evaded with more clever manipulations in order to bury the manipulation signal in noise or something similar
    • Can you differentiate AI assurance here more explicitly from AI governance or audit literature? Seems that these all fall into very nearby categories but neither literature / bodies of work are mentioned nor compared against
  • Parts of the paper read a bit overly casual, which is fine, but sometimes it feels like it interrupts the flow of the paper a little bit (mostly in the introduction)
  • The 4/5ths rule is introduced but honestly, I wasn’t aware of it, though it does seem to be a guideline in the US for content moderation (based on looking this up), could you add a citation for this? In general, I felt citations were often missing (for example, for IRM as well)
1 comment
AuthorsOpenReview ↗

New updates to paper

Thank you for the review and suggestions. We have updated our paper.

Addressing Critical Weaknesses

Thanks for the feedback. We made updates to the paper to help address some weaknesses and gently push back against others. Details below.

*We understand the disappointment that we did not address other AI assurance claims in this paper, but there is much to say about the particular manipulation detection claim on which we focus. Including other claims is out of scope and would come at the expense of a deeper dive on this problem. At the same time, we believe that the tool is more broadly useful in assurance and therefore want to communicate this so others working in AI assurance, who may not previously have considered DML, can build on this. We also have replicated our results on private data but unfortunately cannot include them in the paper.

*Thanks for the baseline suggestions. To clarify, we actually have four baselines in total, two from fairness and two from statistics. We updated the paper to better explain what they are and why we chose them (Section 4.2). We agree it would be interesting to see how DML compares to other causal approaches; though, on the other hand, DML already is a well established causal inference method. Further, none of these other causal or fairwashing methods have ever been applied to our particular problem before. Adding them as baselines is therefore nontrivial and would be a substantial research undertaking for each one.

*We updated the paper with Appendix C to describe how we picked the content moderation classifiers and adumbrate here. Frozen embedding models are relatively common in industry especially when dealing with text at scale where LLMs are prohibitively expensive. So the key choice here is a good text embedding model. We chose our particular embedding model based on the leader board of embedding models for text classification tasks (https://huggingface.co/spaces/mteb/leaderboard). Our criteria was to pick the best reasonably sized model that has a commercial friendly licence, which is how we ended up with EmbeddingGemma-300m. More details now in paper.

*As for a deeper dive on when DML/IRM fails, this is a good point and we agree that this would help improve the understanding of DML as a tool in this space. In fact we had run two additional experiments along these lines but to our chagrin they were only partially succesful at making DML fail. We omitted them in the original submission but based on your suggestion have reconsidered and added them to Appendix B. One experiment is about heavy leakage of treatment information into the nuisance variables and the other is about sample/population mismatch.

Questions for authors

*Yes DML can be evaded, though most mechanisms for doing so are not unique to DML and would fool statistical method as well. For example, by providing skewed data distributions or fake data. Coming up with a DML-specific evasion that's unique to DML is certainly possible but quite tricky. One way to do it might be to target the fact that DML needs overlapping propensity scores to reason counterfactually. To do this we would want to target a manipulation axis that is (1) highly learnable by the content moderation features and (2) highly different from the content moderation labels. It's unclear if this is even possible and moreover it would substantially limit the space of possible manipulations. But people are clever and we would not be surprised if someone finds a way to evade DML specifically.

*On the differences between AI assurance, governance, auditing, etc: The boundary lines between these areas are blurry as they overlap substantially. Rather than trying to crisply differentiate them from assurance, we updated the related work to mention them collectively and place our work in this context. We tend to think of AI assurance as a separate (but related) concern from mitigation and policy-making, though some would disagree with us and we don't want to enter the fray. We loosely use the term behavioral to indicate the fact that we're looking at system inputs and outputs and do not necessarily have the ability to intervene on the data and consider counterfactuals directly (making it a bit stricter than a typical black box audit system that might be able to entertain counterfactuals directly rather than through nuisance models).

*Apologies for any hiccups in the writing, we'll keep our eye out for them.

*We updated the paper to include a description of the 4/5ths rule in section 4.2, though we've opted not to put it in the introduction so it doesn't interrupt the flow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.