Skip to content
EA Crux Project

Jan Leike

📋Page Status
Quality:42 (Adequate)⚠️
Importance:25 (Peripheral)
Last edited:2025-12-24 (14 days ago)
Words:893
Backlinks:5
Structure:
📊 0📈 0🔗 0📚 0•58%Score: 2/15
LLM Summary:Biographical overview of Jan Leike's career trajectory from DeepMind to OpenAI's Superalignment team to Anthropic, focusing on his RLHF work and departure from OpenAI over safety prioritization concerns. Provides qualitative assessment of his views on alignment urgency and scalable oversight without quantitative data or citations.
Researcher

Jan Leike

Importance25
Websiteanthropic.com
RoleHead of Alignment
Known ForAlignment research, scalable oversight, RLHF
Related
Safety Agendas
Organizations
Researchers

Background

Section titled “Background”

Jan Leike is a leading AI alignment researcher currently serving as Head of Alignment at Anthropic. He has a PhD from Australian National University, where he worked on AI safety under Marcus Hutter.

His career has been defined by practical, empirical approaches to alignment:

  • Early work on safe exploration in reinforcement learning
  • Pioneering research on learning from human feedback
  • Leadership of alignment teams at DeepMind, OpenAI, and now Anthropic
  • Focus on scalable methods that can work with current ML paradigms

Career Trajectory

Section titled “Career Trajectory”

DeepMind (2017-2021)

Section titled “DeepMind (2017-2021)”

Worked on the first implementations of learning from human feedback, including:

  • Safe exploration methods
  • Reward modeling
  • Scalable agent alignment

OpenAI (2021-2024)

Section titled “OpenAI (2021-2024)”
  • Joined to lead alignment research
  • Co-led the Superalignment team (announced July 2023)
  • Secured 20% of OpenAI’s compute for alignment research
  • Departed May 2024 over disagreements about safety prioritization

Anthropic (2024-present)

Section titled “Anthropic (2024-present)”
  • Joined as Head of Alignment
  • Reunited with former OpenAI colleagues
  • Leading alignment research on Claude and future systems

Key Contributions

Section titled “Key Contributions”

RLHF Pioneer

Section titled “RLHF Pioneer”

Jan was one of the early researchers to demonstrate that reinforcement learning from human feedback (RLHF) could work at scale:

  • Co-authored seminal papers on reward learning
  • Showed how to train language models to be helpful and harmless
  • Methods he developed became standard across industry

Scalable Oversight Research

Section titled “Scalable Oversight Research”

Core focus on how to supervise AI systems more capable than humans:

  • Recursive reward modeling
  • AI-assisted human evaluation
  • Process supervision vs. outcome supervision
  • Weak-to-strong generalization

Superalignment Vision

Section titled “Superalignment Vision”

At OpenAI, co-led (with Ilya Sutskever) the Superalignment team, which aimed to:

  • Solve alignment for superintelligent systems
  • Use AI to help align even more capable AI
  • Achieve this within four years
  • Dedicate significant compute resources to the problem

Views on Key Cruxes

Section titled “Views on Key Cruxes”
📊Jan Leike's Risk Assessment

Based on public statements and research priorities

SourceEstimateDate
Alignment urgencyVery high2024
Timeline pressureNext 3-5 years critical2024
Technical tractabilityDifficult but solvable2024

Alignment urgency: Left OpenAI over concerns about insufficient safety prioritization

Timeline pressure: Emphasized need to solve alignment soon

Technical tractability: Optimistic about scalable oversight approaches

Core Beliefs

Section titled “Core Beliefs”
  1. Alignment is urgent: We have limited time to solve this before transformative AI
  2. Scalable oversight is key: Central challenge is supervising superhuman AI
  3. Empirical work is essential: Need to test alignment techniques on increasingly capable systems
  4. Safety must be prioritized: Cannot let capability research consistently outpace safety
  5. Current methods are insufficient: RLHF and similar techniques won’t scale to superintelligence without major improvements

Why He Left OpenAI

Section titled “Why He Left OpenAI”

In May 2024, Jan departed OpenAI and posted on X (Twitter):

  • “Building smarter-than-human machines is an inherently dangerous endeavor”
  • “Over the past years, safety culture and processes have taken a backseat to shiny products”
  • Expressed concern about compute and priority allocation for safety

This departure, along with Ilya Sutskever’s, raised significant questions about OpenAI’s commitment to safety research.

Research Focus

Section titled “Research Focus”

Current Priorities at Anthropic

Section titled “Current Priorities at Anthropic”
  1. Weak-to-strong generalization: How can weaker systems (including humans) effectively supervise stronger ones?
  2. Scalable oversight techniques: Making human feedback work for superhuman systems
  3. Honest AI systems: Ensuring AI systems accurately report their reasoning and limitations
  4. Automated alignment research: Using AI to help solve alignment

Key Technical Problems

Section titled “Key Technical Problems”

Jan has identified several crucial challenges:

  • Reward hacking: Systems optimizing proxies rather than true objectives
  • Distributional shift: Maintaining alignment in novel situations
  • Deceptive alignment: Preventing systems from appearing aligned while pursuing other goals
  • Superalignment: Aligning systems smarter than humans

Public Communication

Section titled “Public Communication”

Jan is known for:

  • Clear, technical communication about alignment challenges
  • Willingness to raise concerns publicly
  • Engagement on Twitter/X about safety issues
  • Focus on concrete, actionable research directions

His departure from OpenAI sparked significant public discussion about AI safety prioritization at major labs.

Strategic Views

Section titled “Strategic Views”

On AI Development

Section titled “On AI Development”
  • Safety must keep pace: Capability advances should be matched by safety advances
  • Need serious compute: Alignment research requires significant computational resources
  • Coordination is important: Labs should share safety insights
  • Race dynamics are dangerous: Competition that sacrifices safety is unacceptable

On Research Approach

Section titled “On Research Approach”
  • Empirical and theoretical: Need both practical testing and conceptual work
  • Learn from current systems: Can make progress by studying existing models
  • Prepare for qualitative jumps: Current techniques may not suffice for superintelligence
  • Automate alignment work: Use AI to scale up alignment research itself

Influence and Impact

Section titled “Influence and Impact”

Research Impact

Section titled “Research Impact”
  • RLHF work influenced every major language model deployment
  • Scalable oversight framework guides significant research programs
  • Superalignment vision shaped discourse on superintelligence alignment

Field Building

Section titled “Field Building”
  • Mentored numerous alignment researchers
  • Built and led multiple alignment teams
  • Raised profile of alignment research within major labs

Institutional Influence

Section titled “Institutional Influence”
  • Secured major compute allocation for alignment at OpenAI
  • Helped shape Anthropic’s research priorities
  • Demonstrated importance of independent safety research

Key Publications

Section titled “Key Publications”
  • “Deep Reinforcement Learning from Human Preferences” (2017) - Early RLHF paper
  • “Scalable agent alignment via reward modeling” (2018) - Reward learning framework
  • “Recursively Summarizing Books with Human Feedback” (2021) - Demonstrating RLHF scaling
  • Various blog posts on alignment challenges and approaches

Current Challenges

Section titled “Current Challenges”

At Anthropic, Jan faces several key challenges:

  1. Time pressure: Transformative AI may arrive soon, requiring rapid progress
  2. Scaling RLHF: Current techniques may not work for superintelligent systems
  3. Evaluation: How to know if alignment techniques actually work
  4. Automation: Using AI to help solve alignment before it becomes too capable
  5. Coordination: Ensuring insights are shared across safety community
Section titled “Related Pages”

What links here