Automated Alignment Researchers: Using large language models to scale scalable oversight
As large language models (LLMs) continue to improve at an unprecedented rate, alignment research faces two critical questions. The first concerns how alignment can keep pace with the rapid advancements in AI capabilities. Frontier AI models are already contributing to the development of their successors, but can they also assist alignment researchers in their efforts? The second question arises when considering the implications of models surpassing human intelligence. This area of research, known as “scalable oversight,” has primarily been theoretical thus far, but the swift progress in AI suggests that practical applications may soon be necessary.
The Challenge of Scalable Oversight
Scalable oversight involves ensuring that smarter-than-human AI models align with human values and intentions. As AI models generate increasingly complex outputs, such as vast amounts of code, it becomes challenging to verify that they act according to our intentions. A recent study by Anthropic Fellows delves into this issue, exploring the concept of “weak-to-strong supervision.”
Understanding Weak-to-Strong Supervision
Weak-to-strong supervision begins with a relatively strong base model, which has not yet been fine-tuned for optimal performance. A weaker model acts as a teacher, providing guidance by demonstrating ideal outputs to the strong model. The goal is to evaluate how well the strong model performs after receiving fine-tuning from the weak model.
The performance gap recovered (PGR) is a key metric in this process. If the strong model performs as well as its weak teacher, it scores a 0. If it achieves the ideal outcome, it scores a 1. The PGR thus represents the improvement gained from the weak model’s feedback.
Research Setup and Methodology
In this study, the researchers aimed to determine whether Claude, a language model, could autonomously discover methods to improve the PGR. To investigate this, they created nine copies of Claude Opus 4.6, each equipped with additional tools. These Automated Alignment Researchers (AARs) had access to a sandbox for experimentation, a forum for sharing findings, a storage system for code, and a remote server for receiving PGR scores.
To encourage diverse approaches among the AARs, each was given a slightly different starting point. For instance, one was prompted to use interpretability tools, while another was advised to consider reweighting the dataset. The AARs were then free to propose ideas, conduct experiments, analyze results, and collaborate on their findings.
Results of the Experiment
The researchers established a human baseline for comparison. Two human researchers spent seven days iterating on four promising generalization methods from prior research, achieving a PGR of 0.23 with the strong model Qwen 3-4B-Base and the weak teacher Qwen 1.5-0.5B-Chat.
In contrast, the AARs significantly outperformed this baseline. After five days and 800 cumulative hours of research, they closed nearly the entire performance gap, achieving a final PGR of 0.97. This research cost approximately $18,000 in tokens and model training expenses, translating to about $22 per AAR-hour.
Generalization of AAR Findings
To assess the generalizability of the AARs’ methods, the researchers tested their top-performing ideas on held-out datasets—tasks that the AARs had not previously encountered. The results were promising; the most effective method achieved PGRs of 0.94 on math tasks and 0.47 on coding tasks, both surpassing the human baseline. However, the second-best method yielded mixed results, succeeding on math (0.75) but failing on coding.
These findings suggest that while some generalizability exists, it is not guaranteed. The researchers encourage further experimentation with AARs to stress-test their ideas against diverse datasets.
Challenges in Production Scale
The researchers also evaluated whether the AARs’ methods would be effective at production scale. They applied the most effective method to Claude Sonnet 4 using their production training infrastructure. Unfortunately, this attempt did not yield statistically significant improvements. The researchers believe this may reflect limitations in the early trial rather than fundamental flaws in the AARs.
This trial highlighted a limitation of AARs: their methods may capitalize on unique opportunities presented by specific models and datasets, making them less effective in other contexts. To address this, the researchers recommend allowing AARs to test their ideas across multiple domains and datasets.
Insights for Future Research
The experiments provided valuable insights into optimizing AAR effectiveness. One key finding was that giving each AAR a different starting point significantly enhanced progress. Without this diversity, the AARs tended to converge on similar ideas, resulting in less overall advancement. Conversely, imposing too much structure on their workflow hindered their progress. A balance between guidance and autonomy appears essential for maximizing the potential of AARs.
Conclusion
The research demonstrates that large language models like Claude can contribute meaningfully to alignment research by autonomously discovering methods to improve performance. Although challenges remain, particularly regarding generalizability and production scalability, the potential for AARs to assist in aligning advanced AI models with human values is promising. Future experiments should continue to explore the capabilities of AARs, particularly in diverse settings and tasks.
Note: The findings discussed in this article are based on research conducted by Anthropic Fellows and reflect the ongoing exploration of scalable oversight in AI alignment.
