AI Risk Portfolio Analysis

📋Page Status

Page Type:ContentStyle Guide →Standard knowledge base article

Quality:64 (Good)⚠️

Importance:82 (High)

Last edited:2025-12-28 (6 months ago)

Words:2.2k

Backlinks:2

Structure:

📊 24📈 3🔗 38📚 14•1%Score: 15/15

LLM Summary:Quantitative portfolio framework recommending AI safety resource allocation: 40-70% to misalignment, 15-35% to misuse, 10-25% to structural risks, varying by timeline. Based on 2024 funding analysis ($110-130M total), identifies specific gaps including governance (underfunded by $15-20M), agent safety ($7-12M gap), and international capacity ($11-16M gap).

Issues (3):

• QualityRated 64 but structure suggests 100 (underrated by 36 points)
• Links6 links could use <R> components
• StaleLast edited 170 days ago - may need review

TODOs (4):

• TODOComplete 'Conceptual Framework' section
• TODOComplete 'Quantitative Analysis' section (8 placeholders)
• TODOComplete 'Strategic Importance' section
• TODOComplete 'Limitations' section (6 placeholders)

Model

AI Risk Portfolio Analysis

Importance82

Model TypePrioritization Framework

FocusResource Allocation

Key OutputRisk magnitude comparisons and allocation recommendations

Related

Models

• Compounding Risks Analysis Model
• Flash Dynamics Threshold Model

Model Quality

Novelty

4.5

Rigor

6

Actionability

7.5

Completeness

7.5

Overview

This framework provides quantitative estimates for allocating limited resources across AI risk categories. Based on expert surveys and risk assessment methodologies from organizations like RAND↗🔗 web★★★★☆RAND CorporationRANDRAND conducts policy research analyzing AI's societal impacts, including potential psychological and national security risks. Their work focuses on understanding AI's complex im...governancecybersecurityprioritizationresource-allocation+1Source ↗Notes and Center for Security and Emerging TechnologyOrganizationCSET (Center for Security and Emerging Technology)CSET is a $100M+ Georgetown center with 50+ staff conducting data-driven AI policy research, particularly on U.S.-China competition and export controls. The center conducts hundreds of annual gover...Quality: 43/100 (CSET)↗🔗 web★★★★☆CSET GeorgetownCSET: AI Market DynamicsI apologize, but the provided content appears to be a fragmentary collection of references or headlines rather than a substantive document that can be comprehensively analyzed. ...prioritizationresource-allocationportfolioescalation+1Source ↗Notes, the analysis estimates misalignment accounts for 40-70% of existential risk, misuse 15-35%, and structural risks 10-25%.

The model draws from portfolio optimization theory↗🔗 webportfolio optimization theoryprioritizationresource-allocationportfolioSource ↗Notes and Coefficient Giving’sOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed $4B+ in grants since 2014, including $336M to AI safety (~60% of external funding). The organization spent ~$50M on AI safety in 2024, w...Quality: 55/100 cause prioritization framework↗🔗 webOpen Philanthropy's cause prioritization frameworkprioritizationresource-allocationportfolioSource ↗Notes, addressing the critical question: How should the AI safety community allocate its $100M+ annual resources across different risk categories? All estimates carry substantial uncertainty (±50% or higher), making the framework’s value in relative comparisons rather than precise numbers.

Risk Assessment Matrix

Risk Category	X-Risk Share	P(Catastrophe)	Tractability	Neglectedness	Current Allocation
Misalignment	40-70%	15-45%	2.5/5	3/5	≈50%
Misuse	15-35%	8-25%	3.5/5	4/5	≈25%
Structural	10-25%	5-15%	4/5	4.5/5	≈15%
Accidents (non-X)	5-15%	20-40%	4.5/5	2.5/5	≈10%

Uncertainty Bounds

These estimates represent informed speculation based on limited data. Superforecasters↗🔗 webGood Judgment OpenGood Judgment Open is an online forecasting platform where users can predict future events and compete to become 'Superforecasters'. The platform is operated by Good Judgment, a...prioritizationresource-allocationportfoliohuman-ai-interaction+1Source ↗Notes and AI experts show significant disagreement↗🔗 web★★★☆☆AI ImpactsAI experts show significant disagreementprioritizationresource-allocationportfoliointerventions+1Source ↗Notes on these parameters, with confidence intervals often spanning 2-3x.

Strategic Prioritization Framework

Expected Value Calculation

The framework applies standard expected value methodology:

$$\text{Priority Score} = \text{Risk Magnitude} \times \text{P(Success)} \times \text{Neglectedness Multiplier}$$

Category	Risk Magnitude	P(Success)	Neglectedness	Priority Score
Misalignment	8.5/10	0.25	0.6	1.28
Misuse	6.0/10	0.35	0.8	1.68
Structural	4.5/10	0.40	0.9	1.62

Timeline-Dependent Allocation

Resource allocation should vary significantly based on AGI timeline beliefsConceptAGI TimelineComprehensive synthesis of AGI timeline forecasts showing dramatic acceleration: expert median dropped from 2061 (2018) to 2047 (2023), Metaculus from 50 years to 5 years since 2020, with current p...Quality: 59/100:

Timeline Scenario	Misalignment	Misuse	Structural	Rationale
Short (2-5 years)	70-80%	15-20%	5-10%	Only time for direct alignment work
Medium (5-15 years)	50-60%	25-30%	15-20%	Balanced portfolio approach
Long (15+ years)	40-50%	20-25%	25-30%	Time for institutional solutions

Loading diagram...

Marginal Value Analysis

Current Bottlenecks by Risk Category

Category	Primary Bottleneck	Marginal $ Value	Saturation Risk	Key Organizations
Misalignment	Conceptual clarity	High (if skilled)	Medium	MIRIOrganizationMIRIComprehensive organizational history documenting MIRI's trajectory from pioneering AI safety research (2000-2020) to policy advocacy after acknowledging research failure, with detailed financial da...Quality: 50/100, AnthropicLabAnthropicComprehensive profile of Anthropic, founded in 2021 by seven former OpenAI researchers (Dario and Daniela Amodei, Chris Olah, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish) with early funding...Quality: 51/100
Misuse	Government engagement	Very High	Low	CNAS↗🔗 web★★★★☆CNASCNASagenticplanninggoal-stabilityprioritization+1Source ↗Notes, CSET↗🔗 web★★★★☆CSET GeorgetownCSET: AI Market DynamicsI apologize, but the provided content appears to be a fragmentary collection of references or headlines rather than a substantive document that can be comprehensively analyzed. ...prioritizationresource-allocationportfolioescalation+1Source ↗Notes
Structural	Framework development	High	Very Low	GovAILab ResearchGovAIGovAI is an AI policy research organization with ~15-20 staff, funded primarily by Coefficient Giving ($1.8M+ in 2023-2024), that has trained 100+ governance researchers through fellowships and cur...Quality: 43/100, CAISLab ResearchCAISCAIS is a research organization that has distributed $2M+ in compute grants to 200+ researchers, published 50+ safety papers including benchmarks adopted by Anthropic/OpenAI, and organized the May ...Quality: 42/100
Accidents	Implementation gaps	Medium	High	Partnership on AI↗🔗 webPartnership on AIA nonprofit organization focused on responsible AI development by convening technology companies, civil society, and academic institutions. PAI develops guidelines and framework...foundation-modelstransformersscalingsocial-engineering+1Source ↗Notes

Funding Landscape Analysis

Based on comprehensive analysis from Coefficient GivingOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed $4B+ in grants since 2014, including $336M to AI safety (~60% of external funding). The organization spent ~$50M on AI safety in 2024, w...Quality: 55/100, Longview Philanthropy estimates, and LTFF reporting, external AI safety funding reached approximately $110-130M in 2024:

Funding Source	2024 Amount	Share	Key Focus Areas
Coefficient GivingOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed $4B+ in grants since 2014, including $336M to AI safety (~60% of external funding). The organization spent ~$50M on AI safety in 2024, w...Quality: 55/100	$63.6M	≈49%	Technical alignment, evaluations, governance
Survival & Flourishing Fund	$19M+	≈15%	Diverse safety research
Long-Term Future Fund	$5.4M	≈4%	Early-career, small orgs
Jaan Tallinn & individual donors	$20M	≈15%	Direct grants to researchers
Government (US/UK/EU)	$32.4M	≈25%	Policy-aligned research
Other (foundations, corporate)	$10-20M	≈10%	Various

The breakdown by research area reveals significant concentration in interpretability and evaluations:

Research Area	2024 Funding	Share	Trend	Optimal (Medium Timeline)
Interpretability	$52M	40%	Growing	30-35%
Evaluations/benchmarking	$23M	18%	Rapid growth	15-20%
Constitutional AI/RLHF	$38M	29%	Stable	25-30%
Governance/policy	$18M	14%	Underfunded	20-25%
Red-teaming	$15M	12%	Growing	10-15%
Agent safety	$8.2M	6%	Emerging	10-15%

Coefficient Giving Dominance

Coefficient GivingOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed $4B+ in grants since 2014, including $336M to AI safety (~60% of external funding). The organization spent ~$50M on AI safety in 2024, w...Quality: 55/100 accounts for nearly 60% of all external AI safety investment, with $63.6M deployed in 2024. Since 2017, Coefficient Giving has donated approximately $336M to AI safety (~12% of their total $2.8B in giving). The median Coefficient Giving AI safety grant is $257k; the average is $1.67M.

Risk Interdependency Network

Rather than independent categories, risks exhibit complex interactions affecting prioritization:

Loading diagram...

Correlation Matrix

Risk Pair	Correlation	Implication for Portfolio
Misalignment ↔ Capabilities	+0.8	High correlation; capabilities research affects risk
Misuse ↔ Governance Quality	-0.6	Good governance significantly reduces misuse
Structural ↔ All Others	+0.4	Structural risks amplify other categories

Comparative Assessment Methods

Expert Survey Results

Multiple surveys reveal substantial disagreement on AI risk magnitude. AI Impacts 2022 expert survey↗🔗 web★★★☆☆AI ImpactsAI experts show significant disagreementprioritizationresource-allocationportfoliointerventions+1Source ↗Notes of 738 AI researchers and the Conjecture internal survey provide contrasting perspectives:

Risk Category	AI Impacts Median	Conjecture Median	Expert Disagreement (IQR)	Notes
Total AI X-risk	5-10%	80%	2-90%	Massive disagreement
Misalignment-specific	25%	60%+	10-50%	Safety org workers higher
Misuse (Bio/weapons)	15%	30-40%	5-35%	Growing concern
Economic Disruption	35%	50%+	20-60%	Most consensus
Authoritarian Control	20%	40%	8-45%	Underexplored

Interpreting Survey Disagreement

The Conjecture survey (N=22 AI safety researchers) found no respondent reported less than 10% extinction risk, with a median of 80%. However, this sample has severe selection bias—AI safety researchers self-select for high risk estimates. The AI Impacts survey sampled mainstream ML researchers with lower risk estimates but suffered from non-response bias. True uncertainty likely spans 2-50% for catastrophic outcomes.

Case Study Comparisons

Historical technology risk portfolios provide calibration:

Technology	Primary Risk Focus	Secondary Risks	Outcome Assessment
Nuclear weapons	Accident prevention (60%)	Proliferation (40%)	Reasonable allocation
Climate change	Mitigation (70%)	Adaptation (30%)	Under-weighted adaptation
Internet security	Technical fixes (80%)	Governance (20%)	Under-weighted governance

Pattern: Technical communities systematically under-weight governance and structural interventions.

Uncertainty Analysis

Key Cruxes Affecting Allocation

Key Questions (5)

• What's the probability of transformative AI by 2030? (affects all allocations)
• How tractable is technical alignment with current approaches?
• Does AI lower bioweapons barriers by 10x or 1000x?
• Are structural risks primarily instrumental or terminal concerns?
• What's the correlation between AI capability and alignment difficulty?

Sensitivity Analysis

Parameter Change	Effect on Misalignment Priority	Effect on Misuse Priority
Timeline -50% (shorter)	+15-20 percentage points	-5-10 percentage points
Alignment tractability +50%	-10-15 percentage points	+5-8 percentage points
Bioweapons risk +100%	-5-8 percentage points	+10-15 percentage points
Governance effectiveness +50%	-3-5 percentage points	+8-12 percentage points

Geographic Distribution of Funding

The AI safety funding landscape shows significant geographic concentration, with implications for portfolio diversification:

Region	2024 Funding	Share	Key Organizations	Gap Assessment
SF Bay Area	$48M	37%	CHAI, MIRI, Anthropic	Well-funded
London/Oxford	$32M	25%	FHI, DeepMind, GovAI	Well-funded
Boston/Cambridge	$12M	9%	MIT, Harvard	Growing
Washington DC	$8M	6%	CSET, CNAS, Brookings	Policy focus
Rest of US	$10M	8%	Academic dispersed	Moderate
Europe (non-UK)	$8M	6%	Berlin, Zurich hubs	Underfunded
Asia-Pacific	$4M	3%	Singapore, Australia	Severely underfunded
Rest of World	$8M	6%	Various	Very limited

Emerging Hubs

Government initiatives are expanding geographic coverage: Canada’s $12M AI Safety Research Initiative, Australia’s $8.4M Responsible AI Program, and Singapore’s $5.6M AI Ethics Research Fund launched in 2024-2025. These represent opportunities for funding diversification beyond the US/UK axis.

Loading diagram...

Implementation Recommendations

For Major Funders

Based on 2024 funding analysis, specific portfolio rebalancing recommendations:

Funder Type	Current Allocation	Recommended Shift	Specific Opportunities	Priority
Coefficient GivingOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed $4B+ in grants since 2014, including $336M to AI safety (~60% of external funding). The organization spent ~$50M on AI safety in 2024, w...Quality: 55/100	68% evals, 12% interp	+15% governance, +10% agent safety	GovAI expansion, international capacity	High
SFF/individual donors	Technical focus	+$5-10M to neglected areas	Value learning, formal verification	High
LTFF	Early career, small orgs	Maintain current portfolio	Continue diversified approach	Medium
Government agencies	Policy-aligned research	+$20-30M to independent oversight	AISI expansion, red-teaming	Very High
Tech philanthropists	Varies widely	Coordinate via giving circles	Reduce duplication	Medium

Specific Funding Gaps (2025):

Gap Area	Current Funding	Optimal	Gap	Recommended Recipients
Agent safety	$8.2M	$15-20M	$7-12M	METR, Apollo, academic groups
Value alignment theory	$6.5M	$12-15M	$5-9M	MIRI, academic philosophy
International capacity	$4M	$15-20M	$11-16M	Non-US/UK hubs
Governance research	$18M	$25-35M	$7-17M	GovAI, CSET, Brookings
Red-teaming	$15M	$20-25M	$5-10M	Independent evaluators

For Research Organizations

Capability-Building Priorities:

Organization Size	Primary Focus	Secondary Focus	Rationale
Large (>50 people)	Maintain current specialization	Add governance capacity	Comparative advantage
Medium (10-50 people)	70% core competency	30% neglected areas	Diversification benefits
Small (<10 people)	Focus on highest neglectedness	None	Resource constraints

For Individual Researchers

Career decision framework based on 80,000 Hours methodology↗🔗 web★★★☆☆80,000 Hours80,000 Hours methodologyprioritizationresource-allocationportfoliotalent+1Source ↗Notes:

Career Stage	If Technical Background	If Policy Background	If Economics/Social Science
Early (0-5 years)	Alignment research	Misuse prevention	Structural risk analysis
Mid (5-15 years)	Stay in alignment vs. pivot	Government engagement	Institution design
Senior (15+ years)	Research leadership	Policy implementation	Field coordination

Current State and Trajectory

2024 Funding Landscape

Based on detailed analysis and Coefficient GivingOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed $4B+ in grants since 2014, including $336M to AI safety (~60% of external funding). The organization spent ~$50M on AI safety in 2024, w...Quality: 55/100 grant data, external AI safety funding has evolved significantly:

Year	External Funding	Internal Lab Safety	Total (Est.)	Key Developments
2020	$40-60M	$50-100M	$100-160M	Coefficient Giving ramping up
2021	$60-80M	$100-200M	$160-280M	Anthropic founded
2022	$80-100M	$200-400M	$280-500M	ChatGPT launch
2023	$90-120M	$400-600M	$490-720M	Major lab investment
2024	$110-130M	$500-700M	$610-830M	Government entry

Internal vs External Funding

Major AI labs—Anthropic, OpenAI, and DeepMind—invest an estimated $500M+ combined in internal safety research annually, dwarfing external philanthropic funding. However, internal research may face conflicts of interest with commercial objectives, making external independent funding particularly valuable for governance and red-teaming work.

Coefficient Giving Technical AI Safety Grants (2024)

Detailed analysis of Coefficient Giving’sOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed $4B+ in grants since 2014, including $336M to AI safety (~60% of external funding). The organization spent ~$50M on AI safety in 2024, w...Quality: 55/100 $28M in Technical AI Safety grants reveals:

Focus Area	Share of CG TAIS	Key Recipients	Assessment
Evaluations/benchmarking	68%	METR, Apollo, UK AISI	Heavily funded
Interpretability	12%	Anthropic, Redwood	Well-funded
Robustness	8%	Academic groups	Moderate
Value alignment	5%	MIRI, academic	Underfunded
Field building	5%	MATS, training programs	Adequate
Other approaches	2%	Various	Exploratory

Projected 2025-2027 Needs

Scenario	Annual Need	Technical	Governance	Field Building	Rationale
Short timelines (2-5y)	$300-500M	70%	20%	10%	Maximize alignment progress
Medium timelines (5-15y)	$200-350M	55%	30%	15%	Build institutions + research
Long timelines (15+y)	$150-250M	45%	35%	20%	Institutional capacity

Coefficient Giving’sOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed $4B+ in grants since 2014, including $336M to AI safety (~60% of external funding). The organization spent ~$50M on AI safety in 2024, w...Quality: 55/100 2025 RFP commits at least $40M to technical AI safety, with potential for “substantially more depending on application quality.” Priority areas marked include agent safety, interpretability, and evaluation methods.

Key Model Limitations

What This Framework Doesn’t Capture

Limitation	Impact on Recommendations	Mitigation Strategy
Interaction effects	Under-estimates governance value	Weight structural risks higher
Option value	May over-focus on current priorities	Reserve 10-15% for exploration
Comparative advantage	Ignores organizational fit	Apply at implementation level
Black swan risks	May miss novel risk categories	Regular framework updates

Confidence Intervals

Estimate	90% Confidence Interval	Source of Uncertainty
Misalignment share	25-80%	Timeline disagreement
Current allocation optimality	±20 percentage points	Tractability estimates
Marginal value rankings	Medium confidence	Limited empirical data

Sources & Resources

Funding Data Sources

Source	Type	Coverage	Update Frequency	URL
Coefficient GivingOrganizationCoefficient GivingCoefficient Giving (formerly Open Philanthropy) has directed $4B+ in grants since 2014, including $336M to AI safety (~60% of external funding). The organization spent ~$50M on AI safety in 2024, w...Quality: 55/100 Grants Database	Primary	All CG grants	Real-time	openphilanthropy.org
EA Funds LTFF Reports	Primary	LTFF grants	Quarterly	effectivealtruism.org
Longview Philanthropy Analysis	Analysis	Landscape overview	Annual	EA Forum
CG Technical Safety Analysis	Analysis	CG TAIS breakdown	Annual	LessWrong
Open Philanthropy↗🔗 webOpen Philanthropyprioritizationresource-allocationportfolioSource ↗Notes	Annual reports	Strategy & priorities	Annual	openphilanthropy.org

Expert Surveys

Survey	Sample	Year	Key Finding	Methodology Notes
Grace et al. (AI Impacts)↗🔗 web★★★☆☆AI ImpactsAI experts show significant disagreementprioritizationresource-allocationportfoliointerventions+1Source ↗Notes	738 ML researchers	2022	5-10% median x-risk	Non-response bias concern
Conjecture Internal Survey	22 safety researchers	2023	80% median x-risk	Selection bias (safety workers)
FLI AI Safety Index	Expert composite	2025	24 min to midnight	Qualitative assessment

Academic Literature

Category	Key Papers	Organization	Relevance
Portfolio Theory	Markowitz (1952)↗📄 paperMarkowitz (1952)prioritizationresource-allocationportfolioSource ↗Notes	University of Chicago	Foundational framework
Risk Assessment	Kaplan & Garrick (1981)↗📄 paperKaplan & Garrick (1981)prioritizationresource-allocationportfolioSource ↗Notes	UCLA	Risk decomposition
AI Risk Surveys	Grace et al. (2022)↗🔗 web★★★☆☆AI ImpactsAI experts show significant disagreementprioritizationresource-allocationportfoliointerventions+1Source ↗Notes	AI Impacts	Expert elicitation
MIT AI Risk Repository	MIT	2024	Risk taxonomy

Policy Organizations

Organization	Focus Area	Key Resources	2024 Budget (Est.)
RAND Corporation↗🔗 web★★★★☆RAND CorporationRAND: AI and National Securitycybersecurityagenticplanninggoal-stability+1Source ↗Notes	Defense applications	National security risk assessments	$5-10M AI-related
CSET↗🔗 web★★★★☆CSET GeorgetownCSET: AI Market DynamicsI apologize, but the provided content appears to be a fragmentary collection of references or headlines rather than a substantive document that can be comprehensively analyzed. ...prioritizationresource-allocationportfolioescalation+1Source ↗Notes	Technology policy	AI governance frameworks	$8-12M
CNAS↗🔗 web★★★★☆CNASCNASprioritizationresource-allocationportfolioSource ↗Notes	Security implications	Military AI analysis	$3-5M AI-related
Frontier Model Forum	Industry coordination	AI Safety Fund ($10M+)	$10M+

Related Models

This framework connects with several other analytical models:

• Compounding Risks AnalysisModelCompounding Risks Analysis ModelMathematical framework quantifying how AI risks compound beyond additive effects through four mechanisms (multiplicative probability, severity multiplication, defense negation, nonlinear effects), ...Quality: 60/100 - How risks interact and amplify
• Critical Uncertainties FrameworkCruxCritical Uncertainties ModelIdentifies 35 high-leverage uncertainties in AI risk across compute (scaling breakdown at 10^26-10^30 FLOP), governance (10% P(US-China treaty by 2030)), and capabilities (autonomous R&D 3 years aw...Quality: 71/100 - Key unknowns affecting strategy
• Capability-Alignment Race ModelAnalysisCapability-Alignment Race ModelQuantifies the capability-alignment race showing capabilities currently ~3 years ahead of alignment readiness, with gap widening at 0.5 years/year driven by 10²⁶ FLOP scaling vs. 15% interpretabili...Quality: 62/100 - Timeline dynamics
• Defense in Depth ModelModelDefense in Depth ModelMathematical framework showing independent AI safety layers with 20-60% individual failure rates can achieve 1-3% combined failure, but deceptive alignment creates correlations (ρ=0.4-0.5) that inc...Quality: 69/100 - Multi-layered risk mitigation

Related Pages

What links here

• Worldview-Intervention Mappingmodel
• Intervention Timing Windowsmodel