AI Governance and Policy
AI Governance and Policy
Overview
Section titled “Overview”AI governance encompasses institutions, regulations, and coordination mechanisms designed to shape AI development and deployment for safety and benefit. Unlike technical AI Safety research that solves problems directly, governance creates guardrails, incentives, and coordination mechanisms to reduce catastrophic risk through policy interventions.
This field has rapidly expanded following demonstrations of large language model capabilities and growing concerns about AGI timelines. The Centre for the Governance of AI↗ estimates governance interventions could reduce x-risk by 5-25% if international coordination succeeds, making it potentially one of the highest-leverage approaches to AI safety.
Recent developments demonstrate increasing political momentum: the EU AI Act↗ entered force in 2024, the US Executive Order on AI↗ mandated compute reporting thresholds, and industry Responsible Scaling Policies↗ now cover most frontier labs. However, binding international coordination remains elusive.
Risk/Impact Assessment
Section titled “Risk/Impact Assessment”| Dimension | Assessment | Quantitative Estimate | Confidence |
|---|---|---|---|
| Tractability | Medium | 30-50% chance of meaningful regulation by 2027 in major jurisdictions | Medium |
| Resource Allocation | Growing rapidly | ~$100M/year globally on AI governance research and advocacy | High |
| Field Size | Expanding | ~500-1000 dedicated professionals globally, growing 20-30% annually | Medium |
| Political Will | Increasing | 70%+ of G7 countries have active AI governance initiatives | High |
| Estimated X-Risk Reduction | Substantial if coordinated | 5-25% reduction potential from governance approaches | Low |
| Timeline Sensitivity | Critical | Effectiveness drops sharply if deployed after AGI development | High |
Key Arguments for AI Governance
Section titled “Key Arguments for AI Governance”Coordination Problem Resolution
Section titled “Coordination Problem Resolution”Even perfect technical solutions for AI alignment may fail without governance mechanisms. The racing dynamics problem requires coordination to prevent a “race to the bottom” where competitive pressures override safety considerations. Toby Ord’s analysis↗ suggests international coordination has historically prevented catastrophic outcomes from nuclear weapons and ozone depletion.
Evidence:
- Nuclear Test Ban Treaty reduced atmospheric testing by >95% after 1963
- Montreal Protocol eliminated 99% of ozone-depleting substances
- But success rate for arms control treaties is only ~40% according to RAND Corporation analysis↗
Information Asymmetry Correction
Section titled “Information Asymmetry Correction”AI companies possess superior information about their systems’ capabilities and risks. OpenAI’s GPT-4 System Card↗ revealed concerning capabilities only discovered during testing, highlighting the need for external oversight and mandatory disclosure requirements.
Key mechanisms:
- Pre-deployment testing requirements
- Third-party evaluation access
- Whistleblower protections
- Capability assessment reporting
Market Failure Addressing
Section titled “Market Failure Addressing”Safety is a public good that markets under-provide due to externalized costs. Dario Amodei’s analysis↗ notes that individual companies cannot capture the full benefits of safety investments, creating systematic under-investment without regulatory intervention.
Major Intervention Areas
Section titled “Major Intervention Areas”1. International Coordination
Section titled “1. International Coordination”International coordination aims to prevent destructive competition between nations through treaties, institutions, and shared standards.
Recent Progress:
The Bletchley Declaration↗ (November 2023) achieved consensus among 28 countries on AI risks, followed by the Seoul AI Safety Summit↗ where frontier AI companies made binding safety commitments. The Partnership for Global Inclusivity on AI↗ involves 61 countries in governance discussions.
Proposed Institutions:
- International AI Safety Organization (IAISO): Modeled on IAEA, proposed by Yoshua Bengio and others↗
- UN AI Advisory Body: Interim report published↗ September 2024
- Compute Governance Framework: Lennart Heim’s research↗ proposes international compute monitoring
X-risk reduction if binding international AI governance established
| Source | Estimate | Date |
|---|---|---|
| Centre for the Governance of AI | 20-40% | — |
| RAND Corporation analysis | 15-30% | — |
| FHI technical report | 10-50% | — |
Centre for the Governance of AI: Based on historical arms control success rates and AI-specific factors
RAND Corporation analysis: Factors in verification challenges and compliance incentives
FHI technical report: Wide range reflecting deep uncertainty about implementation success
Key Challenges:
- US-China tensions: Trade war and technology competition complicate cooperation
- Verification complexity: Unlike nuclear weapons, AI capabilities are software-based and harder to monitor
- Enforcement mechanisms: International law lacks binding enforcement for emerging technologies
- Technical evolution: Rapid AI progress outpaces slow treaty negotiation processes
Organizations working on this:
- Centre for the Governance of AI↗ (Oxford)
- Center for Security and Emerging Technology↗ (Georgetown)
- Center for New American Security↗ (CNAS)
- UN Office of the High Representative for Disarmament Affairs↗
2. National Regulation
Section titled “2. National Regulation”National governments are implementing comprehensive regulatory frameworks with legally binding requirements.
United States Framework:
The Executive Order on Safe, Secure, and Trustworthy AI↗ (October 2023) established:
- Compute reporting threshold: Models using >10²⁶ floating-point operations must report to government
- NIST AI Safety Institute: $200M budget↗ for evaluation capabilities
- Pre-deployment testing: Required for dual-use foundation models
Congressional action includes the CREATE AI Act↗, proposing $2.4B for AI research infrastructure, and various algorithmic accountability bills.
European Union AI Act:
The EU AI Act↗ (entered force August 2024) creates the world’s most comprehensive AI regulation:
| Risk Category | Requirements | Penalties |
|---|---|---|
| Prohibited AI | Ban on social scoring, emotion recognition in schools | Up to €35M or 7% global revenue |
| High-Risk AI | Conformity assessment, risk management, human oversight | Up to €15M or 3% global revenue |
| GPAI Models (>10²⁵ FLOP) | Systemic risk evaluation, incident reporting | Up to €15M or 3% global revenue |
| GPAI Models (>10²⁶ FLOP) | Adversarial testing, model cards, code of conduct | Up to €15M or 3% global revenue |
Implementation timeline extends to 2027, with €400M budget↗ for enforcement.
United Kingdom Approach:
The UK AI Safety Institute↗ focuses on pre-deployment testing and international coordination rather than prescriptive regulation. Key initiatives include:
- Capability evaluations: Testing frontier models before public release
- Safety research: £100M funding for alignment and evaluation research
- International hub: Coordinating with US AISI and other national institutes
Other National Developments:
- China: Draft measures↗ for algorithmic recommendation and generative AI regulation
- Singapore: Model AI Governance Framework↗ for voluntary adoption
- Canada: Proposed Artificial Intelligence and Data Act↗ in Parliament
3. Industry Standards and Self-Regulation
Section titled “3. Industry Standards and Self-Regulation”Industry-led initiatives aim to establish safety norms before mandatory regulation, with mixed effectiveness.
Responsible Scaling Policies (RSPs):
Anthropic’s RSP↗ pioneered the IF-THEN framework:
- IF capabilities reach defined threshold (e.g., autonomous replication ability)
- THEN implement corresponding safeguards (e.g., enhanced containment)
Current adoption:
- Anthropic: ASL-2 containment for current models, ASL-3 planned for future systems
- OpenAI: Preparedness Framework↗ with risk assessment scorecards
- Google DeepMind: Frontier Safety Framework↗ for responsible deployment
- Meta: System-level safety approach↗ focusing on red-teaming
Effectiveness Assessment:
- Strengths: Rapid implementation, industry buy-in, technical specificity
- Weaknesses: Voluntary nature, competitive pressure, limited external oversight
Voluntary Safety Commitments:
Post-Seoul Summit commitments↗ from 16 leading AI companies include:
- Publishing safety frameworks publicly
- Sharing safety research with governments
- Enabling third-party evaluation access
Safety-washing concerns↗ highlight the risk of superficial compliance without substantive safety improvements.
Views on whether voluntary commitments can prevent AI catastrophe
4. Compute Governance
Section titled “4. Compute Governance”Compute governance↗ leverages the concentrated, trackable nature of AI training infrastructure to implement upstream controls.
Current Mechanisms:
Export Controls: The October 2022 semiconductor restrictions↗ limited China’s access to advanced AI chips:
- NVIDIA A100/H100 exports restricted to China
- Updated controls (October 2023)↗ closed loopholes
- Estimated to delay Chinese frontier AI development by 1-3 years according to CSET analysis↗
Compute Thresholds:
- EU AI Act: 10²⁵ FLOP threshold for enhanced obligations
- US Executive Order: 10²⁶ FLOP reporting requirement
- UK consideration: Similar thresholds for pre-deployment testing
Proposed Mechanisms:
- Hardware registration: Mandatory tracking of high-performance AI chips
- Cloud compute monitoring: Know-your-customer requirements for large training runs
- International verification: IAEA-style monitoring of frontier AI development
Limitations:
- Algorithmic efficiency gains: Reducing compute requirements for equivalent capabilities
- Distributed training: Splitting computation across many smaller systems
- Semiconductor evolution: New architectures may circumvent current controls
5. Liability and Legal Frameworks
Section titled “5. Liability and Legal Frameworks”Legal liability mechanisms aim to internalize AI risks and create accountability through courts and regulatory enforcement.
Emerging Frameworks:
Algorithmic Accountability:
- EU AI Liability Directive↗ (proposed) creates presumptions of causality
- US state-level algorithmic auditing requirements (e.g., NYC Local Law 144↗)
Product Liability Extension:
- Treating AI systems as products subject to strict liability
- California SB 1001↗ proposed manufacturer liability for AI harms
- Challenge: Establishing causation chains in complex AI systems
Whistleblower Protections:
- EU AI Act Article 85↗ protects AI whistleblowers
- Proposed US federal legislation for AI safety disclosures
- Industry resistance due to competitive sensitivity concerns
Current State & Trajectory
Section titled “Current State & Trajectory”Regulatory Implementation Timeline
Section titled “Regulatory Implementation Timeline”| Jurisdiction | Current Status | 2025 Milestones | 2027 Outlook |
|---|---|---|---|
| EU | AI Act in force, implementation beginning | High-risk AI requirements active | Full enforcement with penalties |
| US | Executive Order implementation ongoing | Potential federal AI legislation | Comprehensive regulatory framework |
| UK | AISI operational, light-touch approach | Pre-deployment testing routine | Possible binding requirements |
| China | Sectoral regulations expanding | Generative AI rules mature | Comprehensive AI law likely |
Industry Compliance Readiness
Section titled “Industry Compliance Readiness”Anthropic’s compliance analysis↗ estimates:
- Large labs: 70-80% ready for EU AI Act compliance by 2025
- Smaller developers: 40-50% ready, may exit EU market
- Open-source community: Unclear compliance pathway for foundation models
International Coordination Progress
Section titled “International Coordination Progress”Achieved:
- Regular AI Safety Summit process established
- Voluntary industry commitments from major labs
- Technical cooperation between national AI Safety Institutes
Pending:
- Binding international agreements on AI development restrictions
- Verification and enforcement mechanisms
- China-US cooperation beyond technical exchanges
Key Uncertainties and Cruxes
Section titled “Key Uncertainties and Cruxes”Technical Feasibility Cruxes
Section titled “Technical Feasibility Cruxes”❓Key Questions
Geopolitical Coordination Cruxes
Section titled “Geopolitical Coordination Cruxes”The central uncertainty is whether US-China cooperation on AI governance is achievable. Graham Allison’s analysis↗ of the “Thucydides Trap” suggests structural forces make cooperation difficult, while Joseph Nye argues↗ shared existential risks create cooperation incentives.
Evidence for cooperation possibility:
- Both countries face AI Risk from uncontrolled development
- Nuclear arms control precedent during Cold War tensions
- Track 1.5 dialogue continuing through official channels
Evidence against cooperation:
- AI viewed as strategic military technology
- Current trade war and technology restrictions
- Domestic political pressure against appearing weak
Timing and Sequence Cruxes
Section titled “Timing and Sequence Cruxes”The relationship between governance timeline and AGI development critically affects intervention effectiveness:
If AGI arrives before governance maturity (3-7 years):
- Focus on emergency measures: compute caps, development moratoria
- International coordination becomes crisis management
- Higher risk of poorly designed but rapidly implemented policies
If governance has time to develop (7+ years):
- Opportunity for evidence-based, iterative policy development
- International institutions can mature gradually
- Lower risk of governance mistakes harming beneficial AI development
Key Organizations and Career Paths
Section titled “Key Organizations and Career Paths”Leading Research Organizations
Section titled “Leading Research Organizations”Academic Institutes:
- Centre for the Governance of AI↗ (Oxford): ~25 researchers, leading governance research
- Center for Security and Emerging Technology↗ (Georgetown): ~40 staff, China expertise and technical analysis
- Stanford Human-Centered AI Institute↗: Policy research and government engagement
- Belfer Center↗ (Harvard Kennedy School): Technology and national security focus
Think Tanks:
- Center for New American Security↗: Defense and technology policy
- Brookings Institution↗: AI governance and regulation analysis
- RAND Corporation↗: Policy analysis and government consulting
- Center for Strategic and International Studies↗: Technology competition and governance
Government Bodies
Section titled “Government Bodies”National AI Safety Institutes:
- US NIST AI Safety Institute↗: ~100 planned staff, $200M budget
- UK AI Safety Institute↗: ~50 staff, pre-deployment testing focus
- EU AI Office↗: AI Act implementation and enforcement
Advisory Bodies:
- US AI Safety and Security Board↗: Private-public coordination
- UK AI Council↗: Industry and academic advice
- EU High-Level Expert Group on AI↗: Ethics and governance guidance
Career Pathways
Section titled “Career Pathways”Entry Level (0-3 years experience):
- Research Assistant at governance organization ($50-70K)
- Government fellowship programs (TechCongress↗, AAAS Science & Technology Policy Fellowships↗) ($80-120K)
- Policy school (MPP/MPA) with AI focus ($80-150K debt typical)
Mid-Level (3-8 years experience):
- Policy researcher at think tank ($80-120K)
- Government policy analyst (GS-13/14, $90-140K)
- Advocacy organization program manager ($90-150K)
Senior Level (8+ years experience):
- Government senior advisor/policy director ($150-200K)
- Think tank research director ($180-250K)
- International organization leadership ($200-300K)
Useful Backgrounds:
- Law (especially administrative, international, technology law)
- Political science/international relations
- Economics (mechanism design, industrial organization)
- Technical background with policy interest
- National security/foreign policy experience
Complementary Interventions
Section titled “Complementary Interventions”AI governance works most effectively when combined with:
- Technical AI Safety Research: Provides feasible safety requirements for regulation
- AI Safety Evaluations: Enables objective capability and safety assessment
- AI Safety Field Building: Develops governance expertise pipeline
- Corporate AI Safety: Ensures private sector implementation of public requirements
- Public AI Education: Builds political support for governance interventions
Risks and Limitations
Section titled “Risks and Limitations”Governance Failure Modes
Section titled “Governance Failure Modes”Premature Lock-in:
- Poorly designed early regulations could entrench suboptimal approaches
- Example: EU’s GDPR complexity potentially serving as template for AI regulation
- Mitigation: Sunset clauses, regular review requirements, adaptive implementation
Regulatory Capture:
- Incumbent AI companies could shape rules to favor their positions
- OpenAI’s advocacy for licensing↗ potentially creates barriers to competitors
- Mitigation: Multi-stakeholder input, transparency requirements, conflict-of-interest rules
Innovation Suppression:
- Overly restrictive regulations could slow beneficial AI development
- Open-source AI development particularly vulnerable to compliance costs
- Mitigation: Risk-based approaches, safe harbors for research, impact assessments
Authoritarian Empowerment:
- AI governance infrastructure could facilitate surveillance and control
- China’s social credit system demonstrates risks of AI-enabled authoritarianism
- Mitigation: Democratic oversight, civil liberties protections, international monitoring
International Coordination Challenges
Section titled “International Coordination Challenges”Free Rider Problem:
- Countries may benefit from others’ safety investments while avoiding costs
- Similar to climate change cooperation difficulties
- Potential solution: Trade linkages, conditional cooperation mechanisms
Verification Difficulties:
- Unlike nuclear weapons, AI capabilities are primarily software-based
- Detection of violations requires access to proprietary code and training processes
- Possible approaches: Hardware monitoring, whistleblower incentives, technical cooperation agreements
Critical Assessment and Evidence Base
Section titled “Critical Assessment and Evidence Base”Track Record Analysis
Section titled “Track Record Analysis”Historical precedents for technology governance:
- Nuclear Non-Proliferation Treaty: 191 signatories, but ~10 nuclear weapons states
- Chemical Weapons Convention: 193 parties, largely effective enforcement
- Biological Weapons Convention: 183 parties, but verification challenges remain
- Montreal Protocol: 198 parties, successful phase-out of ozone-depleting substances
Success factors from past agreements:
- Clear verification mechanisms
- Economic incentives for compliance
- Graduated response to violations
- Technical assistance for implementation
AI governance unique challenges:
- Dual-use nature of AI technology
- Rapid pace of technological change
- Diffuse development across many actors
- Difficulty of capability verification
Current Effectiveness Evidence
Section titled “Current Effectiveness Evidence”| Intervention | Measurable Outcomes | Assessment |
|---|---|---|
| EU AI Act implementation | 400+ companies beginning compliance programs | Early stage, full impact unclear |
| US compute reporting thresholds | 6 companies reported to NIST as of late 2024 | Good initial compliance |
| Export controls on China | ~70% reduction in advanced chip exports to China | Effective short-term, adaptation ongoing |
| Voluntary industry commitments | 16 major labs adopted safety frameworks | High participation, implementation quality varies |
| AI Safety Institute evaluations | ~10 frontier models evaluated pre-deployment | Establishing precedent for external review |
Resource Requirements and Cost-Effectiveness
Section titled “Resource Requirements and Cost-Effectiveness”Global governance investment estimate: $200-500M annually across all organizations and governments
Potential impact if successful:
- 5-25% reduction in existential risk from AI
- Billions in prevented accident costs
- Improved international stability and cooperation
Cost per unit risk reduction:
- Roughly $10-100M per percentage point of x-risk reduction
- Compares favorably to other longtermist interventions
- But high uncertainty in both costs and effectiveness
Getting Started in AI Governance
Section titled “Getting Started in AI Governance”Immediate Actions
Section titled “Immediate Actions”For Policy Students/Early Career:
- Apply to AI Safety Fundamentals Governance Track↗
- Read core papers from Centre for the Governance of AI↗
- Follow policy developments via Import AI Newsletter↗, AI Policy & Governance Newsletter↗
- Apply for fellowships: TechCongress↗, CSET Research↗
For Experienced Professionals:
- Transition via AI Policy Entrepreneurship↗ program
- Engage with Partnership on AI↗ working groups
- Contribute expertise to NIST AI Risk Management Framework↗ development
- Join professional networks: AI Policy Network↗, governance researcher communities
Skills Development Priorities
Section titled “Skills Development Priorities”High-priority skills:
- Policy analysis and development
- International relations and diplomacy
- Technical understanding of AI capabilities
- Stakeholder engagement and coalition building
- Regulatory design and implementation
Medium-priority skills:
- Economics of technology regulation
- Legal framework analysis
- Public communication and advocacy
- Cross-cultural competency (especially US-China relations)
Related Interventions and Cross-Links
Section titled “Related Interventions and Cross-Links”What links here
- Compute Governancepolicy
Key related approaches:
- AI Control - Technical methods that complement governance oversight
- Scheming - Risk that governance aims to prevent through monitoring
- Racing Dynamics - Core problem governance seeks to solve
- Compute Governance - Specific technical governance approach
- AI Safety Field Building - Develops governance expertise pipeline
The effectiveness of AI governance ultimately depends on successful integration across technical safety research, policy development, international coordination, and industry implementation. While individual governance interventions face significant challenges, the coordinated deployment of multiple governance approaches may provide humanity’s best opportunity to navigate advanced AI development safely.
AI Transition Model Context
Section titled “AI Transition Model Context”AI governance improves the Ai Transition Model across multiple factors:
| Factor | Parameter | Impact |
|---|---|---|
| Civilizational Competence | International Coordination | Treaties and coordination mechanisms reduce racing dynamics |
| Civilizational Competence | Regulatory Capacity | National frameworks establish oversight and enforcement |
| Civilizational Competence | Institutional Quality | New institutions like AI Safety Institutes build governance capacity |
| Transition Turbulence | Racing Intensity | Coordinated policies reduce competitive pressure on safety |
Governance interventions are particularly critical for scenarios where technical alignment alone is insufficient and coordination problems require institutional solutions.