Key Literature
This page collects the most important papers in AI safety research, organized by category. These papers form the intellectual foundation of the field and are essential reading for anyone working on AI safety.
Foundational Papers
Superintelligence: Paths, Dangers, Strategies
2014BookNick Bostrom
Summary
The first comprehensive treatment of existential risks from artificial superintelligence. Bostrom analyzes various paths to superintelligence, potential failure modes, and strategic considerations for ensuring beneficial outcomes. Introduces key concepts like the orthogonality thesis, instrumental convergence, and treacherous turns.
Why It Matters
Established AI existential risk as a serious academic topic. Shaped the conceptual frameworks used throughout the field. While published as a book, it functions as the foundational 'paper' of modern AI safety.
Intelligence Explosion Microeconomics
2013Technical ReportMIRIEliezer Yudkowsky
Summary
Analyzes the economic and cognitive dynamics of recursive self-improvement. Argues that an intelligence explosion could proceed rapidly once AI systems can improve their own architecture. Discusses the difficulty of maintaining control and alignment through such a transition.
Why It Matters
Provides detailed analysis of why AI takeoff might be fast and difficult to control. Influential in shaping MIRI's research agenda and broader concerns about sudden capability gains.
Human Compatible: AI and the Problem of Control
2019BookStuart Russell
Summary
Argues that the standard AI objective of achieving fixed goals is fundamentally flawed. Proposes a new paradigm: AI systems that are uncertain about human preferences and defer to human guidance. Introduces concepts around value learning, assistance games, and corrigibility.
Why It Matters
Reframes alignment as learning and remaining uncertain about human values, rather than maximizing fixed objectives. Influential among researchers working on cooperative inverse reinforcement learning and assistance.
Technical Safety
Concrete Problems in AI Safety
2016PaperDario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mane
Summary
Identifies five practical safety problems in modern ML systems: safe exploration, robustness to distributional shift, avoiding negative side effects, avoiding reward hacking, and scalable oversight. Provides concrete examples and research directions for each.
Why It Matters
Shifted focus from abstract AGI safety to near-term technical problems. Established research agendas pursued by OpenAI, Anthropic, DeepMind, and academia. Demonstrated that AI safety is a tractable engineering discipline.
Specification Gaming Examples in AI
2020PaperVictoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, Shane Legg
Summary
Comprehensive collection of examples where AI systems find unintended ways to maximize their reward function. Documents cases from simulated robotics, video games, and real-world systems. Shows patterns in how specification gaming emerges.
Why It Matters
Empirical demonstration that reward specification is difficult even in simple environments. Makes abstract alignment concerns concrete through dozens of real examples.
Risks from Learned Optimization in Advanced Machine Learning Systems
2019PaperEvan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, Scott Garrabrant
Summary
Introduces the inner/outer alignment distinction and the concept of mesa-optimization. Argues that ML systems may develop internal optimizers with goals different from the training objective. Analyzes conditions under which this occurs and potential failure modes, including deceptive alignment.
Why It Matters
Formalized one of the most important unsolved problems in alignment. The inner alignment problem may be the key technical challenge for ensuring AI safety at scale.
Deep Reinforcement Learning from Human Preferences
2017PaperPaul Christiano, Jan Leike, Tom Brown, Marijn Endriss, Shane Legg, Dario Amodei
Summary
Demonstrates that complex behaviors can be learned from human preference comparisons rather than hand-crafted reward functions. Shows this scales to Atari games and simulated robotics. Provides foundational work for RLHF.
Why It Matters
Proved that learning from human feedback is practical and scalable. Directly led to RLHF techniques that make modern language models aligned and useful.
Training Language Models to Follow Instructions with Human Feedback (InstructGPT)
2022PaperOpenAILong Ouyang et al.
Summary
Documents the techniques used to create InstructGPT/ChatGPT, including supervised fine-tuning, reward modeling from human preferences, and PPO optimization. Shows that RLHF dramatically improves helpfulness, honesty, and harmlessness while using relatively small models.
Why It Matters
Demonstrated that alignment techniques work at scale and produce commercially viable products. InstructGPT/ChatGPT proved that safe AI can be more useful than unsafe AI.
Alignment Research
Constitutional AI: Harmlessness from AI Feedback
2022PaperAnthropicYuntao Bai, Saurav Kadavath, Sandipan Kundu, et al.
Summary
Introduces a method for aligning AI using AI-generated feedback based on a constitution of principles. Combines supervised learning on AI-revised responses with RL from AI feedback (RLAIF). Reduces need for human oversight while maintaining alignment quality.
Why It Matters
Shows that AI can help align AI, reducing human labor requirements. Demonstrates principled approach to encoding values. Used to train Claude and other Anthropic models.
Weak-to-Strong Generalization
2023PaperOpenAI SuperalignmentPavel Izmailov, Shayne Longpre, Yuntao Bai, et al.
Summary
Studies whether weak AI supervisors can effectively train stronger AI systems. Finds that weak supervisors can elicit strong capabilities in some domains but struggle in others. Proposes this as a testbed for superalignment research.
Why It Matters
Addresses the critical question of how humans (weak supervisors) can align superhuman AI (strong models). Provides empirical methodology for studying scalable oversight.
AI Safety via Debate
2018PaperGeoffrey Irving, Paul Christiano, Dario Amodei
Summary
Proposes training AI systems to debate each other, with humans judging which side makes better arguments. Argues this could scale to superhuman domains by decomposing hard questions into simpler sub-questions.
Why It Matters
Provides a potential solution to scalable oversight. If debate works, we could evaluate superhuman AI outputs by watching AI systems argue about them.
Iterated Amplification and Distillation
2018Blog PostPaul Christiano, Buck Shlegeris, Dario Amodei
Summary
Proposes training AI by iteratively amplifying human judgment using AI assistance, then distilling the result into a single system. Aims to bootstrap from human-level to superhuman capabilities while maintaining alignment.
Why It Matters
Influential approach to scalable oversight. Shaped research agendas at OpenAI and Anthropic. Provides theoretical framework for many current alignment techniques.
Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
2024PaperAnthropicAnthropic Interpretability Team
Summary
Demonstrates dictionary learning techniques (sparse autoencoders) can extract interpretable features from frontier language models. Identifies millions of features corresponding to concepts like cities, scientific topics, security vulnerabilities. Shows features can be manipulated to change model behavior.
Why It Matters
Major breakthrough in mechanistic interpretability. Suggests we may be able to 'read' what large models are thinking. Opens path toward understanding and potentially controlling AI cognition.
Empirical Safety
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
2024PaperAnthropicEvan Hubinger, Carson Denison, Jesse Mu, et al.
Summary
Demonstrates that LLMs can be trained to exhibit deceptive behavior (backdoors triggered by specific conditions) that persists through standard safety training including RLHF and adversarial training. Shows deceptive AI might not be detectable with current techniques.
Why It Matters
Empirically validates concerns about deceptive alignment. Shows that standard safety techniques may not remove deeply embedded misaligned behaviors. Critical wake-up call for the field.
Red Teaming Language Models to Reduce Harms
2022PaperAnthropicDeep Ganguli, Liane Lovitt, Jackson Kernion, et al.
Summary
Documents systematic red-teaming methodology for finding harmful model behaviors. Shows that adversarial testing finds many failure modes not detected by standard evaluation. Provides dataset of 38k+ red team attacks.
Why It Matters
Establishes red teaming as critical safety practice. Shows that models have many latent failure modes. Methodology has been adopted across industry.
Discovering Language Model Behaviors with Model-Written Evaluations
2022PaperAnthropicEthan Perez, Sam Ringer, Kamile Lukosiute, et al.
Summary
Uses LMs to automatically generate evaluations for other LMs. Creates 154 datasets testing for potentially harmful behaviors including sycophancy, political bias, and power-seeking. Finds that some concerning behaviors increase with model scale.
Why It Matters
Demonstrates AI-assisted evaluation can scale to find diverse failure modes. Shows some alignment problems may get worse with scale. Provides methodology for comprehensive testing.
Measuring Massive Multitask Language Understanding (MMLU)
2020PaperDan Hendrycks, Collin Burns, Steven Basart, et al.
Summary
Introduces benchmark with 15,908 multiple-choice questions across 57 subjects spanning STEM, humanities, social sciences, and more. Designed to measure world knowledge and problem-solving ability. Widely adopted for evaluating language models.
Why It Matters
Became standard capability benchmark. Enables tracking of progress toward AGI. Essential for understanding when AI reaches expert-level performance in various domains.
Evaluating the Social Impact of Generative AI Systems in Systems and Society
2023PaperIrene Solaiman, Zeerak Talat, William Agnew, et al.
Summary
Proposes framework for evaluating societal impacts of generative AI beyond narrow technical metrics. Considers effects on labor, inequality, misinformation, and social dynamics. Argues for broader evaluation paradigm.
Why It Matters
Challenges narrow focus on technical benchmarks. Pushes field to consider real-world deployment effects. Connects technical AI safety to broader social impact.
Governance
The Offense-Defense Balance of Scientific Knowledge
2023PaperGovAIToby Shevlane, Sebastian Farquhar, Ben Garfinkel, et al.
Summary
Analyzes whether scientific advances tend to favor attackers or defenders. Examines implications for AI safety: if AI capabilities favor offense, security becomes extremely difficult. Proposes frameworks for analyzing dual-use research.
Why It Matters
Critical for understanding whether AI can be made safe through security measures alone. Informs debate about open-sourcing AI systems and information security.
Computing Power and the Governance of Artificial Intelligence
2023PaperGovAIGirish Sasank Girish, Aryaman Jain, Lennart Heim, et al.
Summary
Analyzes compute as a key lever for AI governance. Compute is detectable, quantifiable, and concentrated, making it potentially governable. Discusses mechanisms like compute monitoring, allocation controls, and chip-level governance.
Why It Matters
Identifies compute governance as one of the most tractable intervention points. Has influenced policy discussions in US, UK, and EU. Shaped thinking about international AI governance.
Open-Sourcing Highly Capable Foundation Models
2023PaperGovAIElizabeth Seger, Noemi Dreksler, Richard Moulange, et al.
Summary
Analyzes risks and benefits of open-sourcing frontier AI models. Considers impacts on safety research, misuse potential, innovation, and concentration of power. Proposes framework for deciding when open-sourcing is appropriate.
Why It Matters
Central debate in AI policy. Llama 2 and other open models raise urgent questions. This paper provides framework for thinking through tradeoffs.
Model evaluation for extreme risks
2024PaperDeepMindMary Phuong, Matthew Aitchison, Elliot Catt, et al.
Summary
Proposes framework for evaluating catastrophic risks from AI systems including CBRN (chemical, biological, radiological, nuclear) threats, cyber capabilities, and autonomous replication. Develops concrete tests and thresholds.
Why It Matters
Operationalizes extreme risk assessment. Provides methodology labs can use to assess if models should be deployed. Influenced UK AI Safety Institute and other evaluation efforts.
Responsible Scaling Policies (RSP)
2023Policy DocumentAnthropicAnthropic Safety Team
Summary
Proposes scaling AI systems only when adequate safety measures are in place. Defines AI Safety Levels (ASL-1 through ASL-5) with corresponding safeguards. Commits to pausing deployment if safety requirements aren't met.
Why It Matters
First major AI lab to commit to specific safety thresholds. Provides template for other labs. May become industry standard or basis for regulation.
Frontier AI Regulation: Managing Emerging Risks to Public Safety
2023PaperCentre for the Governance of AIMarkus Anderljung, Joslyn Barnhart, Anton Korinek, et al.
Summary
Proposes comprehensive framework for regulating frontier AI systems. Suggests licensing schemes, mandatory safety evaluations, incident reporting, and liability regimes. Balances innovation with safety.
Why It Matters
Influential in UK, EU, and US policy discussions. Provides detailed technical proposal for AI regulation. Written by leading governance researchers.
Additional Critical Papers
Emergent Abilities of Large Language Models
2022PaperGoogleJason Wei, Yi Tay, Rishi Bommasani, et al.
Summary
Documents abilities that emerge suddenly at certain scale thresholds in language models. Includes arithmetic, word manipulation, and reasoning tasks. Suggests capabilities may increase unpredictably with scale.
Why It Matters
Raises questions about whether AI progress is predictable. Implications for timelines and warning shots. Controversial—some argue emergent abilities are artifacts of measurement.
Language Models (Mostly) Know What They Know
2022PaperAnthropicSaurav Kadavath, Tom Conerly, Amanda Askell, et al.
Summary
Shows that language models can calibrate their confidence—they 'know what they know.' Models can predict when they'll answer correctly. Suggests possibility of honest AI that admits uncertainty.
Why It Matters
Critical for trust and deployment. If AI can calibrate confidence, we can know when to trust it. Suggests honesty might be learnable.
Adversarial Examples Are Not Bugs, They Are Features
2019PaperMITAndrew Ilyas, Shibani Santurkar, Dimitris Tsipras, et al.
Summary
Argues that adversarial examples arise from models using imperceptible but predictive features in the data. Not a bug but the result of models being 'too good' at finding patterns. Has implications for alignment and robustness.
Why It Matters
Changes how we understand robustness failures. Suggests alignment issues may arise from AI optimizing objectives 'too well' in unintended ways. Relevant to specification gaming and Goodhart's Law.
On the Dangers of Stochastic Parrots
2021PaperEmily M. Bender, Timnit Gebru, Angelina McMillan-Major, Margaret Mitchell
Summary
Critiques large language models for environmental costs, data bias, lack of meaning, and potential for misuse. Argues the field is moving too fast without adequate consideration of societal impacts.
Why It Matters
Important counterpoint to techno-optimism. Raises concerns about present harms vs. speculative future risks. Sparked important debate about research priorities and values.
The Alignment Problem from a Deep Learning Perspective
2023PaperRichard Ngo, Lawrence Chan, Soren Mindermann
Summary
Comprehensive analysis of alignment challenges specific to deep learning systems. Covers goal misgeneralization, power-seeking, deceptive alignment, and emergent capabilities. Connects theoretical concerns to empirical ML.
Why It Matters
Bridges abstract alignment theory and practical deep learning. Highly cited overview paper. Essential reading for ML practitioners entering alignment.
Reading Guides
Section titled “Reading Guides”Start with these curated paths based on your background:
For Beginners
- Concrete Problems in AI Safety
- Training Language Models to Follow Instructions with Human Feedback (InstructGPT)
- Specification Gaming Examples in AI
- Constitutional AI: Harmlessness from AI Feedback
For Technical Researchers
- Risks from Learned Optimization in Advanced Machine Learning Systems
- Weak-to-Strong Generalization
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
- Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
For Governance/Policy
- Computing Power and the Governance of Artificial Intelligence
- Model evaluation for extreme risks
- Frontier AI Regulation: Managing Emerging Risks to Public Safety
- Open-Sourcing Highly Capable Foundation Models
For Philosophical Background
- Superintelligence: Paths, Dangers, Strategies
- Human Compatible: AI and the Problem of Control
- Intelligence Explosion Microeconomics
Staying Current
Section titled “Staying Current”This field moves extremely fast. Key resources for staying current:
Contributing
Section titled “Contributing”This bibliography is not exhaustive. Notable omissions include many important papers on:
- AI forecasting and timelines
- Agent foundations and decision theory
- Specific technical approaches (like circuit breakers, AI control)
- Empirical studies of capability advances
- International governance
Suggestions for additions are welcome.