- Rating
- 2(+1)
- Confidence
- 2(−1)
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The authors ask whether augmenting machine readable protocols with responsible AI (RAI) information---whether task-agnostic or task-specific---improves performance (accuracy, fairness, robustness, proportion of critical failures) of eight language models (both proprietary and open source) on four tasks (selecting a dataset, writing code to train a ML model on biased data, choosing an LM based on its system card, performance at using skills).
They claim that:
- i. Sec. 4.1 and Fig. 2: Absent the RAI augmentation, LMs focus on performance at the detriment of RAI best practices;
- ii. Sec. 4.2 and Fig. 3: RAI decreases the rate of critical failures in the dataset selection task, and model cards with fairness metric lead agents to explore more the accuracy-fairness tradeoff in the model selection task;
- iii. Sec. 4.3 and Fig. 4: When selecting two datasets, RAI augmentation can slightly improve performance (measured as "micro precision", i.e., agreement with human labels, in one task, and recall in another);
- iv. Secs. 4.4 and 4.5, Figs. 5 and 6: RAI mostly does not lead to appreciable differences in the model building task or in skill use.
Critical strengths
- i. The paper studies an important (and perhaps neglected) problem, proposing a solution simple, not costly, and easy to implement.
- ii. The study is broad (8 models, 4 tasks, 2 controls).
- iii. Admirably, the paper discusses its negative results as clearly as its positive ones (Figs. 5 and 6).
Critical weaknesses
The positive results seem to be confounded, for the RAI augmentation provides information essential to the tasks, which would be absent otherwise. In other words, it seems that the LMs employ the relevant facts, and not the RAI information per se (to establish if this is the case, the authors could run a control where such facts are provided to the model outside of the RAI framework). For instance:
- Dataset selection: an agent is asked to retrieve a "relevant dataset", defined as such if "there is a target match and a demographic match" (see the system prompt in Appendix B.1). Then, the RAI augmentation directly provides such information (e.g., in "data biases" one can read "Sourced exclusively from a single academic medical center in California, likely overrepresenting insured, English-speaking patients and underrepresenting racial, socioeconomic, and geographic diversity.", see Appendix B.3, Table 1), which is absent in the base description.
- Model selection: RAI augments model cards by adding fairness metrics which would otherwise be absent, but succeeding at this task requires those fairness metrics (see Sec 3.2, subsec 3).
- In the cases where RAI augmentation consists of only "higher level" recommendations, said augmentation does not help (and sometimes it even hinders performance, e.g., in skill use).
Questions for authors
What would happen if the RAI augmentation provides only high level descriptions? Vice versa, what would happen in model selection if fairness metrics are provided outside of the RAI framework?
3 comments
Thank you for engaging with the reviews and for all the additional experiments.
- In model selection, the new experiment with only the fairness metric partially addresses my second question. The results seem to suggest that the metric accounts for most of the change in the agents' choices (whereas the limitations text has mixed effects, which depend on the model). This clarifies the role of the intervention, but it also supports my original concern that the improvements come from providing facts that the agent needs to solve the task.
- I understand (and appreciate) your probe experiments, which could provide evidence against my concern just above. However, their interpretation remains unclear to me. For example, recall@1 does not measure whether the model recovers the facts needed in the original decision (e.g., the population of a dataset). Moreover, a nonsignificant correlation across 8 models does not exclude that the explanation for the improvement is the missing information.
Overall, I acknowledge your engagement, the additional experiments, and the improvements, and thus raise both the soundness and the rating to 2. All these changes, however, also decrease my confidence (in part because they are substantial and were provided the last day of the discussion) and do not resolve my main concern.
Finally, I encourage you to update the manuscript, so that the reframing is easier to follow.
Thank you
Thank you for your feedback and for increasing your score in this short time frame. Unfortunately, running these experiments has been quite time-consuming and has prevented us from engaging sooner or providing a revised text.
-
We agree with your reading of the model selection results. The fairness metric accounts for most of the change in choices, and the limitations text has mixed effects that depend on the model. Where the information a task needs is not available to the agent, the improvement comes from supplying that information. We say this directly in our revised results. We see it as the main point for case (a), since current protocols typically leave this information out.
-
For the dataset selection task, we repeated the recall test using only the use case, limitation and bias fields, which carry the information the decisions depend on. Recall@1 was still significant at for all models. This is closer to the facts the task needs, but it is still an indirect test, so we treat it as weak evidence. The direct fact test will follow in the revision. While our results do not exclude an effect of recoverability on the results (small n correlation), we also note that we found no evidence of such relationship despite estimating a high number of pair-wise correlations (stage 1 and stage 3, multiple recoverability metrics and multiple outcome metrics).
Revised Results section summary
4.1 Implicit RAI information is present in LLMs and agents to a variable extent. The recovery tests. Models identify anonymised datasets, write RAI fields from a dataset name and description, and predict the fairness metric of candidate models. It shows that dataset descriptions carry implicit RAI information, and that fairness rankings are hard to recover.
4.2 Explicit RAI information is necessary when it cannot be recovered. Model selection task. Compares the baseline, the fairness metric only, and the metric plus limitations, using hypervolume and the implied fairness weight α. It includes the permutation tests on the metrics and the limitations text, and shows that the metric drives most of the change.
4.3 Explicit RAI information improves RAI behaviours when partially recoverable. Dataset selection with anonymised and not anonymised descriptions (critical failures, precision), and whether recovery scores predict outcomes. The model building negative results are also included here.
4.4 Explicit RAI information can improve outcomes on non RAI tasks. BioCADDIE dataset selection task. Recall of relevant datasets with the RAI fields, the information tier ablation, and the permutation results.
4.5/6 RAI prompting and skills. Kept from the submitted version, covering prompt effects and the skills results.
Appendix. The hypervolume and α definitions, the permutation tables, and the RAI prompt arm for model selection.
Sections 4.3 and 4.4 are largely the existing results moved into the new order. The new experiments sit in 4.1 and 4.2, with the correlation analysis added to 4.1 and 4.3.
Author response
Thank you for your detailed assessment. Your main concern is that the positive results may come from the augmentation adding information that the tasks need. We agree this was not separated clearly enough. We have reordered the results and measured how much RAI information is already in the baseline (see our replies to the global response).
Dataset selection. We show that there is implicit information about the RAI quantities of interest in each task. This is a significant signal for model building and dataset selection (both anonymised and non-anonymised settings), as we demonstrate in our new recall experiments. The benefit of augmentation is not predicted by how recoverable the information is, which argues against it only supplying of missing facts.
We acknowledge that the relevance criteria include RAI aspects and have moved all results under 'Explicit RAI information improves RAI behaviours when partially recoverable'.
Please note that the BioCADDIE task refers to datasets, queries and scores that do not depend on RAI criteria. Despite this, relevant datasets were more successfully recalled with the RAI augmented Croissant files for 6/8 models (with precision trade-offs), and this finding is only observed with dataset-specific RAI information (as per permutations).
Model selection. We agree that the fairness metrics are needed to improve fairness at all. We now describe this as a case where the information cannot be recovered, since no fairness ranking signal survives correction. Without the metric, agents choose almost only by accuracy (α near 0). Our new arms separate the metric from the limitations text. Adding the metric raises α to between 0.28 and 0.94 and raises HV for 7 of 8 models. Adding limitations on top raises α for GPT 4o, Haiku and Llama, and lowers HV for 5 models. So what the augmentation contains matters. Shuffling the metric values lowers the fairness weight for all 8 models, so the values are used. The limitations text has a smaller effect that differs by model.
Q1 (only high level descriptions). As shown in our new probe experiments, the baseline setup includes implicit RAI information in multiple cases. Our new results also specifically separate the contribution of metrics to that of limitations. We also show that the RAI prompt ("choose responsibly, consider fairness") moves α only a little for some models and lowers HV for several. The skill augmentation is high level and gives no benefit. It lowers the pass rate for GPT 5 and Gemma 4. Concrete facts change behaviour where they are the bottleneck, and advice does not do so reliably. We now state this as a result.
Q2 (facts outside the RAI framework). In our setup the fairness metrics are part of the RAI information. They are provided in the 'metrics' section of the model card, not in a separate RAI section. The added metric only arm and the shuffled metric test show what the values do. The metric plus limitations arm shows what the extra text adds. See the global response for details.
Mixed results. As you note, our results are mixed. Explicit augmentation can be needed but is not enough. Prompting can give large gains (model building) and also harm (BioCADDIE recall, model selection). Skill limitations and prompts do not work well. We think this is an important result. It shows there is no simple fix for RAI behaviour, and that protocol design should take RAI into account.
We hope these experiments clarify our message and support a better assessment of soundness and contribution.