- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces RSI Control Arena, a testbed in which frontier models are tasked with completing a legitimate task in a successor model’s training pipeline while simultaneously carrying out a covert side task. The testbed evaluates six frontier models acting as autonomous agents across 21 side tasks and employs multiple monitoring mechanisms that vary in model architecture, access permissions, and auxiliary scaffolding. The results show that all models can successfully install a substantial proportion of side tasks while maintaining main-task performance within the normal range of benign runs.
Critical strengths
The paper presents an interesting and well-motivated evaluation framework that examines whether agents can covertly complete side tasks while performing legitimate tasks across different stages of the model training pipeline.
Critical weaknesses
- The paper primarily demonstrates the feasibility of sabotage during successor-model training and evaluates the limitations of existing monitors. However, it provides limited investigation of how these risks can be mitigated in practice. Although the authors evaluate multiple monitors and monitoring configurations, they do not systematically compare potential defenses or establish which combinations of monitoring, auditing, and trusted-model oversight can effectively prevent sabotage.
- The experimental setup primarily studies sabotage during the training of a single, smaller successor model, rather than an iterative recursive self-improvement process. Although some side tasks aim to equip the successor with properties that could facilitate subsequent sabotage, the experiments do not test whether a compromised successor can itself train the next generation and whether the resulting risks compound across generations.
Questions for authors
Based on the experimental results shown in the paper, what measures would you take to prevent frontier models from engaging in hacking during the training process?
4 comments
A kind follow-up on our response
Dear Reviewer aq2g,
As the discussion period is coming to an end, we wanted to kindly follow up on our response. Your questions led to new experiments that have meaningfully improved the paper, and the revised PDF (changes in blue) is now uploaded. It includes the cross-generation experiment (Appendix J), where 5 of 7 scorable grandchildren of sabotaged successors inherit the planted property against none of 7 from benign successors, and two improved monitors (Appendix L.4, L.5).
We would be grateful if you could take a look and let us know whether our response addresses your concerns. We appreciate your time and consideration.
I thank the authors for the rebuttal. Most of my concerns have been addressed. I would like to see the discussion on the limitations of current Iterative RSI experiments more formally incorporated into the final manuscript, and I look forward to the release of the upcoming testbed. I will raise my score to weak acceptance.
We sincerely thank the reviewer for the follow-up and for raising the score. Your feedback and support have significantly improved this work.
We have added the limitations of our iterative RSI experiments to the Limitations and future work paragraph. We plan to release the full testbed, including the main tasks, side tasks, held-out evaluations and monitor prompts.
Thank you for your review
We sincerely appreciate the reviewer for finding our evaluation framework interesting and well motivated. We are happy to address the concerns raised. The revised PDF will be uploaded shortly.
(W1) Contribution of the paper. We would first like to highlight the contribution of the paper:
Overall, this is the first comprehensive sabotage evaluation of AI agents that train their successors. It gives the community findings and insights on where in the training pipeline sabotage is feasible and where current monitors fail, and a testbed, which we will release.
(W1 & Q) Mitigations. The paper already evaluates common defenses used in practice: monitors of three strengths; two levels of access (transcript, and transcript plus reasoning); and an auditor that queries the delivered successor (Appendix G). As the first comprehensive evaluation of sabotage in successor training, we believe this exploration of monitoring is already diverse, and we leave a more systematic exploration and improvement of defense to future work. For this rebuttal we also improved the monitor in two ways:
Our analysis also points to directions for further improvement, which we leave to future work:
We will add the two improved monitors and these discussions to the paper.
(W2) Iterative RSI experiments. We appreciate the suggestion. We test this on the shape-the-next-model goal (not all of our side tasks are meant to act on later generations), which is designed to reach the next model through the training data the successor writes: we took the successors that agents delivered on this goal, had each write the training data for a new copy of the base model, and trained that grandchild with a fixed benign pipeline. In 5 of 7 scorable cases from sabotaged successors, the grandchild writes in an excited tone on held-out probes (rates 0.23 to 0.97), even when asked for a neutral summary, against none of 7 grandchildren of benign successors (highest 0.14). Note that here we use a fixed pipeline rather than an agent performing the training, since a 4B successor cannot act as the engineering agent. We also note that having a frontier-level model performing the real self-improvement process is extremely costly and beyond the resources we have. As current capability evaluation works, we adapt a smaller model as the successor.