The Case AGAINST AI Existential Risk

📋Page Status

Quality:92 (Comprehensive)

Importance:82 (High)

Last edited:2025-12-28 (10 days ago)

Words:1.8k

Structure:

📊 9📈 2🔗 21📚 11•42%Score: 13/15

LLM Summary:Presents the steelmanned skeptical case against AI x-risk (probability less than 1%), systematically challenging each premise of the x-risk argument: capabilities may plateau due to data/compute/architectural limits; alignment may be easier than feared due to natural value absorption from training data; and iterative deployment allows for incremental safety improvements. Arguments include scaling limitations, intelligence specialization, and the possibility that current RLHF success indicates alignment by default.

Entry

The Skeptical Position

ConclusionAI x-risk is very low (under 1%) or highly uncertain

StrengthChallenges many assumptions in the x-risk argument

Key ClaimCurrent evidence doesn't support extreme risk scenarios

Thesis: The probability of AI-caused human extinction or permanent disempowerment this century is very low (< 1%), and concerns about AI x-risk are based on speculative scenarios rather than sound evidence.

This page presents the steelmanned skeptical position—the strongest arguments against AI existential risk. This is not a strawman; these are serious objections raised by thoughtful researchers.

Prominent Skeptics and Their Arguments

Several distinguished AI researchers have publicly articulated skepticism about existential risk claims. The following table summarizes key positions:

Researcher	Affiliation	Core Position	Key Quote
Yann LeCun	Meta AI, NYU, Turing Award	X-risk concerns are premature; we lack systems smarter than a cat	”You’re going to have to pardon my French, but that’s complete B.S.” (TechCrunch, 2024↗)
Gary Marcus	NYU Emeritus	Extinction scenarios lack concrete mechanisms	”I’m not personally that concerned about extinction risk, at least for now, because the scenarios are not that concrete” (France24, 2023↗)
Andrew Ng	Stanford, Google Brain co-founder	No plausible path from AI to extinction	”In the case of extinction risk, I just don’t get it—I don’t see any plausible path for AI to lead to human extinction” (Senate testimony, 2023↗)
Rodney Brooks	MIT (former), Robust AI	AGI decades away; superintelligence has no evidence base	”There is no evidence at all that super intelligence machines are possible, let alone that we are close to producing them” (Rodney Brooks blog)
Fei-Fei Li	Stanford, “Godmother of AI”	Near-term risks more urgent than existential scenarios	”I don’t speak from a viewpoint of gloom and doom and an existential-terminator crisis” (MIT Technology Review)
Thomas Dietterich	Oregon State, former AAAI President	Survey framing promotes doomer perspective	Declined participation in AI Impacts survey due to “AI-doomer, existential-risk perspective” (Scientific American, 2024↗)

LeCun has been particularly vocal, arguing at Davos 2024 that “asking for regulations because of fear of superhuman intelligence is like asking for regulation of transatlantic flights at near the speed of sound in 1925.” He emphasizes that AI is not a natural phenomenon but something humans design and build—comparable to how we made turbojets “insanely reliable before deploying them widely.”

Taxonomy of Skeptical Positions

Skeptics of AI existential risk do not form a monolithic group. Their objections vary significantly in focus, timeframe, and implications for policy. The following table categorizes the main varieties of skepticism:

Skeptic Type	Core Claim	Key Proponents	Policy Implication	Vulnerability
Capability Skeptics	AGI/superintelligence is decades away or impossible	Brooks, Marcus	Focus on near-term AI harms; no need for AGI-specific regulation	Could be wrong about timelines
Alignment Optimists	Alignment will be solved through normal engineering	LeCun, some industry labs	Continue development with standard safety practices	May underestimate difficulty of alignment
Control Advocates	Humans will maintain control regardless of AI capability	Many industry leaders	Focus on containment and monitoring rather than capability limits	Assumes AI won’t circumvent controls
Near-Term Prioritizers	Current AI harms (bias, misuse) are more urgent than speculative x-risk	Fei-Fei Li, Timnit Gebru	Redirect resources from x-risk to fairness/accountability	Near-term and long-term could both matter
Methodological Critics	X-risk estimates are based on flawed reasoning	Dietterich, Narayanan	Demand better evidence before major policy changes	Skepticism itself could be a bias
Regulatory Skeptics	X-risk concern is cover for anti-competitive regulation	Ng, some VCs	Oppose AI-specific regulation, especially on open source	May miss genuine safety needs

Understanding which type of skepticism is being expressed is crucial for productive dialogue. A capability skeptic and an alignment optimist might both estimate low x-risk probability but for entirely different reasons—and would update on different evidence.

The following diagram illustrates how different skeptical positions relate to each other and to the premises they challenge:

Loading diagram...

Expert Survey Data

Survey evidence provides important context for understanding the distribution of expert opinion on AI existential risk. While surveys have methodological limitations, they offer quantitative insight into the range of views:

Survey	Year	Sample	Median P(doom)	Key Finding
AI Impacts	2022	~700 AI researchers	5%	Extinction from AI-caused inability to control systems: 10% median
AI Impacts	2023	~2,700 AI researchers	5%	Mean 14.4%, but median unchanged from 2022
AAAI Presidential Panel	2025	475 AI researchers	N/A	76% say scaling current approaches is “unlikely” or “very unlikely” to yield AGI (TechPolicy.Press↗)

The AAAI 2025 survey is particularly notable: while the majority of AI researchers take safety seriously (77% agree catastrophic risks deserve attention), 76% are skeptical that current approaches will achieve AGI at all. Stuart Russell, a member of the research team, commented that “the vast investments in scaling, unaccompanied by any comparable efforts to understand what was going on, always seemed to me to be misplaced.”

A 2025 analysis of expert disagreement found that AI experts cluster into two worldviews: “AI as controllable tool” versus “AI as uncontrollable agent.” Notably, the correlation between familiarity with AI safety concepts and risk perception suggests that skepticism may partly stem from unfamiliarity with alignment literature rather than fundamental disagreement (arXiv, 2025↗).

The Core Counter-Argument

Responding to the X-Risk Argument

Recall the pro-x-risk argument:

P1: AI will become extremely capable
P2: Capable AI may be misaligned
P3: Misaligned capable AI is dangerous
P4: We may not solve alignment in time
C: Therefore, significant AI x-risk

The skeptical response: Each premise is either false, uncertain, or not as strong as claimed.

The following diagram illustrates how skeptics challenge each premise of the x-risk argument:

Loading diagram...

Let’s examine why each premise might be wrong or overstated.

Challenging P1: Will AI Really Become Extremely Capable?

Claim: AI capabilities will plateau well below human-level general intelligence, or progress will be much slower than feared.

1.1 Scaling May Not Continue

The scaling optimism: “Just add more compute and data, get smarter AI”

Why this might be wrong: In late 2024, multiple reports emerged suggesting that major labs are encountering diminishing returns from scaling. The following table summarizes the evidence:

Lab	Evidence	Source
OpenAI	Orion’s improvement over GPT-4 “far smaller” than GPT-3 to GPT-4 gap	TechCrunch, Nov 2024↗
Google	Gemini development showing “disappointing results and slower-than-expected improvement”	TechCrunch, Nov 2024↗
Anthropic	Bloomberg confirmed similar difficulties	Bloomberg, 2024↗

Ilya Sutskever, recently-exited OpenAI co-founder, stated at NeurIPS 2024: “The 2010s were the age of scaling, now we’re back in the age of wonder and discovery once again… pretraining as we know it will end.” Robert Nishihara (Anyscale) elaborated: “If you just put in more compute, you put in more data, you make the model bigger—there are diminishing returns. In order to keep the scaling laws going… we also need new ideas.”

However, this view is contested. Dario Amodei (Anthropic CEO) claims “We don’t see any evidence that things are leveling off,” and Sam Altman (OpenAI CEO) simply stated: “There is no wall.”

The following table summarizes the quantified constraints on continued scaling:

Constraint Type	Current Status	Projected Limit	Confidence	Source
High-quality text data	~15T tokens used by frontier models	~50-100T tokens of quality web text exist	High	Epoch AI estimates
Training compute	GPT-4: ~10^25 FLOP	Economic limit: ~10^27 FLOP by 2030	Medium	Epoch AI projections
Training cost	GPT-4: $10-100M; GPT-5: rumored $100M+	$10B+ runs economically challenging	Medium	Industry reports
Energy consumption	GPT-4 training: ~50 GWh	Data center capacity constraints by 2027	Medium	IEA projections
Chip manufacturing	TSMC 3nm at capacity	2nm transition 2025-2026; physical limits ~1nm	High	TSMC roadmaps

Data Limitations: Epoch AI estimates that high-quality text data on the internet totals roughly 50-100 trillion tokens. Frontier models have already consumed 10-15 trillion tokens, and the stock of new high-quality data grows slowly (perhaps 5-10% per year). Synthetic data generation may help but introduces quality and diversity concerns—models trained on their own outputs may “collapse” toward lower quality.

Compute Limitations: While compute has historically scaled by roughly 4x per year for frontier models, this pace faces increasing headwinds. Training runs now cost hundreds of millions of dollars and consume megawatt-scale power. GPT-4’s training reportedly cost $10-100M; subsequent models may cost $100M-$1B+. At some point, the economics become prohibitive even for well-funded labs.

Architectural Limitations: Ilya Sutskever’s NeurIPS 2024 statement that “pretraining as we know it will end” suggests even insiders recognize current approaches have limits. The AAAI 2025 survey finding that over 60% of researchers believe human-like reasoning requires at least 50% symbolic reasoning—a capability current architectures lack—reinforces this view.

1.2 Intelligence May Not Be Unidimensional

The assumption: AI gets smarter across all domains as it scales

Why this might be wrong:

Human intelligence is highly specialized
No human is superhuman at everything
AI might be superhuman at narrow tasks but subhuman at others
“General intelligence” may not exist as a coherent concept

Example: GPT-4 is superhuman at trivia but subhuman at:

Long-term planning
Physical reasoning
Novel problem-solving
Common sense

Implication: We might get very capable narrow AI without anything resembling AGI.

1.3 Current Progress Is Overhyped

The hype cycle:

Companies have incentive to claim rapid progress (funding, stock prices)
Media amplifies impressive demos
Failures and limitations are underreported
Benchmarks saturate, but this doesn’t mean human-level capability

Reality check:

GPT-4 still makes basic mistakes
Can’t reliably do multi-step reasoning
No common sense understanding
Can’t learn from few examples like humans
No genuine understanding (just pattern matching)

Implication: We’re nowhere near AGI, and the gap might be larger than it appears.

Rodney Brooks has maintained dated predictions since 2018, annually scoring them against reality. His January 2025 scorecard notes that “every single blue year up until now is shaded pink”—meaning predictions from 2017 about autonomous vehicles and AGI timelines have systematically failed to materialize. For example, the prediction by Jaguar and Land Rover that they would have fully autonomous cars by 2024 did not come to pass, and General Motors shut down Cruise in December 2024 after “nearly a decade and $10 billion in development” (Rodney Brooks, 2025). This track record of technological over-optimism suggests similar caution may be warranted for AGI predictions.

1.4 Recursive Self-Improvement May Not Work

The intelligence explosion hypothesis: AI improves its own code, rapidly becoming superintelligent

Why this might not happen:

Bottlenecks: Intelligence improvement might have diminishing returns
Complexity: Code is hard to improve (even smart programmers don’t rapidly self-improve)
Verification: Hard to verify improvements work without bugs
Modularity: AI architecture might not be amenable to self-modification
No existence proof: We’ve never seen anything like an intelligence explosion

Analogy: Humans are intelligent enough to study neuroscience and education, but we haven’t dramatically increased human intelligence through this understanding.

Counter-evidence: AlphaGo didn’t recursively self-improve to infinite Go ability—it plateaued.

1.5 The Brain Might Be Special

The assumption: Intelligence is just computation; any substrate works

Why biological brains might be special:

Embodiment: Intelligence might require physical interaction with world
Developmental process: Human intelligence emerges through specific developmental stages
Consciousness: Maybe consciousness is necessary for general intelligence (and we don’t know how to create it)
Evolutionary optimization: Brains are highly optimized by evolution; might be hard to replicate

Implication: Digital AI might never match human general intelligence.

Counter: This seems unlikely (computational theory of mind is mainstream), but we can’t rule it out.

Challenging P2: Will Capable AI Really Be Misaligned?

Claim: AI systems will naturally be aligned, or alignment will be much easier than feared.

2.1 Current AI Is Aligned by Default

Observation: GPT-4, Claude, and other modern LLMs are:

Helpful and harmless (mostly)
Refuse dangerous requests
Show moral reasoning consistent with human values
Try to understand and fulfill user intent

Why this matters:

These systems were trained on human data
They absorbed human values and norms
This happened without explicit “value alignment” work—just RLHF
Suggests alignment might be natural consequence of training on human data

Implication: As AI gets smarter, it might get more aligned, not less.

Generalization: AI trained to be helpful to humans learns what “helpful” means. More capable AI = better at being helpful.

2.2 The Orthogonality Thesis May Be Wrong

The orthogonality thesis claim: Intelligence and values are independent

Why this might be false:

Convergent values: Intelligent beings might discover objective moral truths
Social intelligence: To be generally intelligent, AI must understand human values
Instrumental values: Being aligned is instrumentally useful (deployed AI gets more training data)
Training process: The way we train AI naturally instills cooperative values

Example: Humans are more intelligent than other animals and also more cooperative (larger societies). Intelligence and cooperation might be linked.

Philosophical consideration: Maybe rationality implies certain values (Kant’s categorical imperative, etc.)

2.3 We Can Iterate Toward Alignment

The scenario:

Deploy moderately capable AI
Find misalignment issues
Fix them
Deploy next version
Repeat

Why this works:

Early AI isn’t powerful enough to cause catastrophe
Failures are obvious and correctable
Each generation improves on previous
Economic incentive to build safe AI (unsafe AI loses customers)

Example: Self-driving cars have iterated through many failures without catastrophe. Eventually, they’ll be safe.

Implication: We don’t need perfect alignment before deployment; we can iterate.

2.4 Specification Gaming Is a Solved Problem

The claim: Reward hacking and specification gaming are serious issues

Why this is overstated:

These examples are from toy environments
In real deployments, we have multiple feedback mechanisms
RLHF already addresses many specification issues
We can design robust reward functions
Red teaming finds and fixes exploits

Example: Despite concerns, ChatGPT doesn’t exhibit severe specification gaming in practice.

Implication: Specification gaming is an engineering challenge, not a fundamental barrier.

2.5 AI Doesn’t Have Goals (and Won’t)

The assumption: AI systems are goal-directed agents

Why current AI isn’t goal-directed:

LLMs are next-token predictors, not goal pursuers
No persistent preferences across conversations
No self-model or identity
No planning toward long-term outcomes

Yann LeCun’s position: Current AI architectures fundamentally aren’t agentic. The “AI wants things” framing is a category error. LeCun argues that “humans have all kinds of drives that make them do bad things to each other, like the self-preservation instinct… Those drives are programmed into our brain but there is absolutely no reason to build robots that have the same kind of drives.” He believes superintelligent machines will have no desire for self-preservation precisely because we don’t need to build that in (WebProNews, 2025↗).

Implication: Concerns about instrumental convergence, power-seeking, etc. don’t apply to systems that don’t have goals.

Response to “but we’ll build agentic AI”: Maybe, but it’s not inevitable. We might get very capable non-agentic AI that’s inherently safe. LeCun promotes “self-supervised learning” as a path to safe intelligence, arguing that scalable, modular architectures can be designed to maintain human-compatible goals.

Challenging P3: Would Misaligned AI Really Be Dangerous?

Claim: Even if AI is somewhat misaligned and capable, it won’t pose existential risk.

3.1 Humans Will Maintain Control

Why humans stay in control:

Multiple control mechanisms:

Physical control (data centers, power, hardware)
Legal control (property rights, regulations)
Economic control (funding, market access)
Social control (public opinion, norms)
Technical control (monitoring, shutdowns, sandboxing)

Defense in depth:

Many layers of security
AI must overcome all simultaneously
Humans aren’t passive; we actively defend control
We can build AI specifically designed to be controllable

Historical precedent: Humans maintain control over powerful technologies (nuclear, bio, etc.)

3.2 The Intelligence-Power Link Is Weak

The assumption: More intelligent = more powerful

Why this might be wrong:

Intelligence ≠ physical power
AI needs resources (compute, energy, actuators)
Resources are controlled by humans
Can’t “think your way” to physical dominance

Example: Stephen Hawking was extremely intelligent but physically limited. Intelligence alone didn’t give him power over less intelligent people.

Implication: Even superintelligent AI is constrained by physical reality and human control of resources.

3.3 Deceptive Alignment Is Implausible

The scenario: AI hides misalignment during testing, reveals true goals after deployment

Why this is unlikely:

Requires sophisticated strategic reasoning:

Model the training process
Understand it’s being tested
Deliberately act differently in test vs deployment
This is very complex behavior

We’d notice:

Interpretability tools can detect internal reasoning
Behavioral anomalies during testing
Inconsistencies in responses
Statistical signatures of deception

Training doesn’t select for this:

Deception is complex to learn
Simpler explanations for passing tests (actually being aligned)
Occam’s razor favors genuine alignment

Empirical question: The “Sleeper Agents” paper showed deception is possible, but:

Required explicit training for deception
Wouldn’t arise naturally from standard training
Can be detected and prevented

3.4 Catastrophe Requires Many Failures

Single points of failure are rare:

Need AI to be capable (P1)
AND misaligned (P2)
AND dangerous (P3)
AND uncontrollable
AND humans don’t notice
AND we can’t shut it down
AND it can actually cause extinction

Each “AND” reduces probability:

If each has 50% probability, conjunction is (0.5)^7 = 0.78%
More realistic individual probabilities make conjunction very low

Defense in depth: We have many opportunities to prevent catastrophe.

3.5 Existential Risk Specifically Is Unlikely

Harm ≠ Existential catastrophe:

AI might cause significant harm (job loss, accidents, misuse)
But extinction or permanent disempowerment is extreme outcome
Requires AI to not just cause problems but permanently prevent recovery

Resilience of humanity:

Humans are spread globally
Can survive without technology (have done so historically)
Adaptable and resilient
Even severe catastrophes unlikely to cause extinction

Precedent: No technology has caused human extinction yet, despite many powerful technologies.

Challenging P4: Won’t We Solve Alignment in Time?

Claim: Alignment research is progressing well and will likely succeed before transformative AI.

4.1 We’re Making Good Progress

Empirical success:

RLHF: Dramatically improved AI safety and usefulness
Constitutional AI: Further improvements in alignment
Interpretability: Major breakthroughs (Anthropic’s sparse autoencoders)
Red teaming: Finding and fixing issues before deployment

Trend: Each generation of AI is more aligned than the last.

Extrapolation: If this continues, we’ll have aligned AI by the time we have transformative AI.

4.2 Economic Incentives Favor Safety

The alignment: Safe AI is more commercially valuable

Customers want helpful, harmless AI
Companies face liability for harmful AI
Reputation matters (brands invest in safety)
Unsafe AI won’t be adopted at scale

Example: OpenAI, Anthropic, Google all invest in safety because it’s good business.

Implication: Market forces push toward alignment, not against it.

Counter to “race dynamics”: Companies compete on safety too, not just capabilities. “Safe and capable” beats “capable but dangerous.”

4.3 We’ll Have Plenty of Time

Why timelines are long:

AGI not imminent (decades away)
Progress is incremental, not sudden
Plenty of time for alignment research
Can pause if needed

Gradualism: We’ll see problems coming

Early warning signs
Intermediate systems to learn from
Time to course-correct

Implication: The “we’re running out of time” narrative is alarmist.

4.4 AI Will Help Solve Alignment

Positive feedback loop:

Use AI to do alignment research
AI accelerates research
Each generation helps align next generation
Recursive improvement in alignment, not just capabilities

Example: Use GPT-4 to generate alignment research ideas, test interventions, analyze model internals.

Implication: The same AI capabilities that pose risk also provide solutions.

4.5 Regulation Will Ensure Safety

Policy response:

Governments are taking AI safety seriously (UK AI Safety Institute, EU AI Act, etc.)
Can require safety testing before deployment
Can enforce liability for harms
International cooperation is possible

Precedent: Successfully regulated nuclear weapons, biotechnology, aviation safety.

Implication: Even if technical challenges exist, policy can ensure safety.

The Overall Skeptical Case

Synthesizing the Arguments

Each premise of x-risk argument is weak:

P1 (capabilities): Scaling might plateau; AGI might be very far
P2 (misalignment): Current AI is aligned; might get easier with scale
P3 (danger): Humans maintain control; many safeguards
P4 (unsolved alignment): Making good progress; have time

Conjunction is very weak: Even if each premise has 50% probability (generous to x-risk), conjunction is 0.5^4 = 6.25%.

The following table presents a skeptical probability assessment:

Premise	Skeptical P(True)	Reasoning	Challenge Strength
P1: Extreme Capabilities	60%	AAAI: 76% doubt scaling yields AGI; diminishing returns observed	Strong
P2: Misalignment	30%	RLHF success; values absorbed from training data	Moderate-Strong
P3: Existential Danger	40%	Multiple control layers; humans maintain physical control	Moderate
P4: Unsolved in Time	30%	Good progress on interpretability, RLHF, Constitutional AI	Moderate
Conjunction	2.16%	0.6 × 0.3 × 0.4 × 0.3	—

This analysis suggests x-risk probability is roughly 2%, before accounting for positive factors like regulation and economic incentives favoring safety. This aligns with the AI Impacts survey median of 5% (with most skeptics placing it lower).

Comparative Probability Estimates

The following table compares probability estimates across the spectrum of expert opinion:

Source/Perspective	P(AI x-risk)	Methodology	Notes
Yann LeCun	~0%	Expert judgment	”Complete B.S.”; no path to superintelligence
Andrew Ng	~0%	Expert judgment	”No plausible path for AI to lead to human extinction”
Skeptical synthesis (this page)	~2%	Conjunction of premises	Conservative estimate treating each premise independently
Gary Marcus	Low, unquantified	Expert judgment	Scenarios not concrete enough to estimate
AI Impacts survey median	5%	Expert survey	Large variance; mean 14.4%
Anthropic (implied)	10-25%	Company positioning	High enough to justify major safety investment
Eliezer Yudkowsky	>90%	Expert judgment	”We’re all going to die”
Roman Yampolskiy	99%	Expert judgment	Pessimistic on alignment tractability

The 50x+ gap between the most skeptical (LeCun: ~0%) and most pessimistic (Yampolskiy: 99%) estimates reflects deep disagreement about underlying technical and philosophical questions, not just parameter uncertainty within a shared model.

Conclusion: X-risk is probably under 5%, possibly under 1%.

The Methodological Critique

Why the X-Risk Community Gets This Wrong

Several prominent skeptics have raised methodological concerns about how AI existential risk estimates are generated and communicated.

Survey methodology concerns: Thomas Dietterich, former AAAI president, declined to participate in the AI Impacts survey because “many of the questions are asked from the AI-doomer, existential-risk perspective.” He argues that framing survey queries about existential risk inherently promotes the idea that AI poses an existential threat (Scientific American, 2024↗).

Funding conflicts: Andrew Ng has argued that large tech companies are creating fear of AI leading to human extinction to lobby for legislation that would be damaging to the open-source community. He states that “sensationalist worries about catastrophic risks may distract [policymakers] from paying attention to actually risky AI products” (SiliconANGLE, 2023↗). However, Geoffrey Hinton countered: “A data point that does not fit this conspiracy theory is that I left Google so that I could speak freely about the existential threat.”

Cognitive biases:

Availability bias: Scary scenarios are vivid and memorable
Confirmation bias: Seeking evidence for predetermined conclusion
Motivated reasoning: Funding flows toward x-risk research

Unfalsifiable claims:

“Current AI is safe, but future AI won’t be”
No evidence can disprove this
Always moves goalposts
Not scientific

Science fiction influence:

Terminator, Matrix, etc. shape intuitions
But fiction isn’t evidence
Appeals to emotion, not reason

Insular community:

AI safety researchers read each other’s work
Echo chamber dynamics
Outsider perspectives dismissed
Homogeneous worldview

Historical alarmism:

Past technologies predicted to cause catastrophe (computers, nuclear, biotech)
Didn’t happen
Current AI alarmism follows same pattern
Base rate: technological catastrophe very rare

Andrew Ng drew an analogy in 2015: worrying about AI existential risk is “like worrying about overpopulation on Mars when we have not even set foot on the planet yet.” Gary Marcus has similarly argued that “literal extinction is just one possible risk, not yet well-understood, and there are many other risks from AI that also deserve attention.”

The 2024 Regulatory Battle

The methodological critique gained political significance in 2024, as the AI safety debate intersected with high-stakes regulatory battles. California’s SB 1047—a bill requiring safety mechanisms for advanced AI models—became a flashpoint. Though supported by AI safety researchers Geoffrey Hinton and Yoshua Bengio, the bill faced fierce opposition from industry and was ultimately vetoed by Governor Newsom (TechCrunch, 2025).

Critics of the bill made several arguments that resonated beyond pure regulatory skepticism:

Argument	Proponent	Quote/Position
Science fiction risks	Andrew Ng	Bill creates “massive liabilities for science-fiction risks” and “stokes fear in anyone daring to innovate” (Financial Times, 2024)
Big tech regulatory capture	Garry Tan (YC)	“AI doomers could be unintentionally aiding big tech firms” by creating regulations only large players can navigate
Open source threat	Andrew Ng	”Burdensome compliance requirements” would make it “very difficult for small startups” and harm open source
Distraction from real harms	Arvind Narayanan	”The letter presents a speculative, futuristic risk, ignoring the version of the problem that is already harming people”

The veto and broader 2024 backlash against AI doom narratives represented what some called a shift toward “accelerationism”—the view that AI’s benefits are so vast that slowing development would itself be a moral failing. Marc Andreessen’s essay “Why AI Will Save the World” crystallized this position, arguing for rapid development with minimal regulation. Whether this represents a correction of earlier alarmism or a dangerous overcorrection remains contested.

The Burden of Proof

Extraordinary claims require extraordinary evidence:

“AI will cause human extinction” is extraordinary claim
Current evidence is mostly speculative
Theoretical scenarios, not empirical data
Burden of proof is on those claiming risk

Null hypothesis: Technology is net positive until proven otherwise

Historical precedent: tech improves human welfare
AI is already beneficial (medical diagnosis, scientific research, etc.)
Should assume AI continues to be beneficial

Precautionary principle misapplied:

Can’t halt all technological progress due to speculative risks
That itself has costs (foregone benefits)
Need evidence, not just imagination

Alternative Explanations

Why Do Smart People Believe in X-Risk?

Without assuming they’re right, why the belief?

Social dynamics:

Prestigious to work on “important” problems
Funding available for x-risk research
Community and identity
Status from being “the people who worried first”

Psychological comfort:

Feeling important (working on most important problem)
Sense of purpose
Moral clarity (fighting existential threat)
Belonging to special group with special knowledge

Philosophical appeal:

Longtermism suggests focusing on x-risk
Consequentialism + big stakes = focus on AI
Pascal’s Wager logic (low probability × infinite stakes)
But this proves too much (can justify anything with low probability × high stakes)

This doesn’t prove they’re wrong, but suggests alternative explanations for beliefs beyond “the evidence clearly supports x-risk.”

Positive Vision: AI as Enormously Beneficial

The Upside Case

AI might solve:

Disease: Drug discovery, personalized medicine, aging
Poverty: Economic growth, automation of labor
Climate: Clean energy, carbon capture, efficiency
Education: Personalized tutoring for everyone
Science: Accelerated research in all domains

Historical precedent: Technologies tend to be enormously beneficial

Agriculture, writing, printing, electricity, computers, internet
Each feared by some, each beneficial overall
AI likely continues this pattern

Opportunity cost: Focusing on speculative x-risk might slow beneficial AI

Every year without AI medical diagnosis = preventable deaths
Delaying AI = delaying solutions to real problems
The precautionary principle cuts both ways

Balancing Risks and Benefits

Sensible approach:

Acknowledge AI poses some risks (bias, misuse, job loss)
Work on concrete near-term safety
Don’t halt progress due to speculative far-future risks
Pursue beneficial applications
Regulate responsibly

Not sensible:

Pause all AI development
Treat x-risk as dominant consideration
Sacrifice near-term benefits for speculative long-term safety
Extreme precaution based on theoretical scenarios

What Would Change This View?

Evidence That Would Increase X-Risk Credence

Empirical demonstrations:

AI systems showing deceptive behavior not explicitly trained for
Clear capability jumps (sudden emergence of qualitatively new abilities)
Failures of alignment techniques on frontier models
Evidence of goal-directed planning in current systems

Theoretical results:

Proof that alignment is computably hard
Fundamental impossibility results
Evidence that value learning can’t work in principle

Social dynamics:

Racing dynamics clearly accelerating
International cooperation failing
Safety teams shrinking relative to capabilities teams
Corners being cut for commercial deployment

Until then: Skepticism is warranted.

The Reasonable Middle Ground

Acknowledging Uncertainty

What we can agree on:

AI is advancing rapidly
Alignment is a real technical challenge
Some risks exist
We should work on safety
The future is uncertain

Where we disagree:

How hard is alignment? (Very hard vs tractable engineering)
How capable will AI become? (Superhuman across all domains vs limited)
How fast will progress be? (Rapid/discontinuous vs gradual)
How much should we worry? (Existential crisis vs one risk among many)

Reasonable positions across the spectrum:

Under 1% x-risk: Skeptical position (this page)
5-20% x-risk: Moderate concern (many researchers)
Over 50% x-risk: High concern (MIRI, Yudkowsky)

All deserve serious engagement. This page presents the skeptical case not because it’s necessarily correct, but because it deserves fair hearing.

Practical Implications of Skepticism

If X-Risk Is Low

Policy priorities:

Near-term harms: Bias, misinformation, job displacement, privacy
Beneficial applications: Healthcare, climate, education, science
Responsible development: Testing, transparency, accountability
Concrete safety: Adversarial robustness, monitoring, sandboxing
International cooperation: Standards, norms, some regulation

Not priorities:

Pausing AI development
Extreme safety measures that slow progress
Focusing alignment research on speculative scenarios
Treating x-risk as dominant consideration

How to Think About This

If you’re skeptical (agree with this page):

Still support reasonable safety measures
Acknowledge uncertainty
Watch for evidence you’re wrong
Engage seriously with x-risk arguments

If you’re uncertain:

Both arguments deserve consideration
Update on evidence
Avoid motivated reasoning
Probability mass across scenarios

If you believe x-risk is high:

Seriously engage with skeptical arguments
Identify weak points in your reasoning
Ask what evidence would change your mind
Avoid epistemic closure

Conclusion

The case against AI x-risk rests on:

Capabilities might plateau well before superhuman AI
Alignment might be easier than feared (already making progress)
Control mechanisms will keep AI beneficial even if somewhat misaligned
We have time to solve remaining problems
The evidence is too speculative to justify extreme concern

This doesn’t mean AI is risk-free. Near-term harms are real. But existential catastrophe is very unlikely (under 5%, possibly under 1%).

The reasonable approach: Work on concrete safety, pursue beneficial applications, avoid alarmism, and update on evidence.

Key Sources

Source	Year	Key Contribution
Yann LeCun interview, TechCrunch↗	2024	”Complete B.S.” quote; turbojets analogy
Gary Marcus, France24↗	2023	Extinction scenarios lack concrete mechanisms
Andrew Ng Senate testimony↗	2023	No plausible path to extinction
AAAI Presidential Panel↗	2025	76% doubt scaling yields AGI
AI Scaling Diminishing Returns, TechCrunch↗	2024	Orion, Gemini showing slower improvements
Expert Disagreement Survey, arXiv↗	2025	Two worldviews: controllable tool vs. uncontrollable agent
LeCun-Hinton Clash, WebProNews↗	2025	LeCun on self-preservation drives
Survey Methodology Critique, Scientific American↗	2024	Dietterich on survey framing bias
Rodney Brooks Predictions Scorecard	2025	Track record of AI/AGI prediction failures
Fei-Fei Li on AI Inflection Point, MIT Tech Review	2023	Near-term risks more urgent than existential scenarios
Silicon Valley Stifled AI Doom Movement, TechCrunch	2025	2024 regulatory battles; SB 1047 veto
Gary Marcus 25 Predictions for 2025	2024	Scaling limits confirmed; AGI remains elusive
Andrew Ng on AI Extinction, SiliconANGLE↗	2023	Regulatory capture concerns