UK AI Safety Institute
Overview
Section titled “Overview”The UK AI Safety Institute (UK AISI) is a government organization established in November 2023 to advance AI safety through research, evaluation, and international coordination. Created in the wake of the first AI Safety Summit at Bletchley Park, AISI represents the UK’s commitment to being a global leader in AI safety and governance. In February 2025, the organization was renamed the AI Security Institute to reflect a sharper focus on national security threats including cyberattacks, fraud, and AI-enabled bioweapon development.
As one of the first government bodies specifically dedicated to advanced AI safety (alongside the US AISI), the UK AISI plays a pioneering role in translating AI safety research into government policy and practice. The organization conducts technical research on AI risks, evaluates frontier models for dangerous capabilities, develops safety standards, and coordinates international efforts through the International Network of AI Safety Institutes, which now includes 10+ member countries.
The UK AISI’s strategic positioning reflects the UK government’s ambition: to be a global hub for AI safety, bridging technical research communities, frontier AI labs, and international policymakers. Chair Ian Hogarth has described AISI as a “startup inside the government” that combines technical rigor with policy influence.
Organization at a Glance
Section titled “Organization at a Glance”| Attribute | Details |
|---|---|
| Founded | November 2023 (following AI Safety Summit) |
| Renamed | February 2025 (to AI Security Institute) |
| Parent Department | Department for Science, Innovation and Technology (DSIT) |
| Leadership | Ian Hogarth (Chair) |
| Technical Staff | 30+ researchers (as of mid-2024) |
| Annual Budget | Approximately 50 million GBP |
| Locations | London (HQ), San Francisco (opened 2024) |
| Key Output | Inspect AI framework, Frontier AI Trends Report |
| International Role | Founding member of International Network of AI Safety Institutes |
History and Founding
Section titled “History and Founding”Background: UK AI Safety Summit (November 2023)
Section titled “Background: UK AI Safety Summit (November 2023)”Bletchley Park Declaration:
- UK hosted first international AI Safety Summit at Bletchley Park
- 28 countries and EU signed declaration on AI risks
- Recognized catastrophic risks from advanced AI
- Committed to international cooperation on safety
- Political moment crystallizing government attention
Summit outcomes:
- Agreement on establishing AI Safety Institutes
- UK and US announced creation of their institutes
- Commitments from frontier AI labs
- International research collaboration agreements
- Foundation for ongoing coordination
Context:
- ChatGPT moment (late 2022) raised AI awareness
- Growing concern about AI existential risk
- UK positioning for leadership role
- Rishi Sunak government prioritizing AI policy
- Academic and researcher advocacy for government action
Establishment (2023)
Section titled “Establishment (2023)”Founding: Announced at AI Safety Summit, operationalized late 2023
Organizational placement:
- Part of UK Department for Science, Innovation and Technology (DSIT)
- Government agency with civil service staff
- Independent research capacity
- Policy coordination role
- International engagement mandate
Initial mission:
- Conduct safety research on advanced AI
- Evaluate frontier models for dangerous capabilities
- Develop testing and evaluation methodologies
- Coordinate international AI safety efforts
- Advise UK government on AI risks and governance
Funding: The UK initially invested 100 million GBP in the Frontier AI Taskforce (AISI’s predecessor), providing more funding for AI safety than any other country at the time. The Institute’s annual budget is approximately 50 million GBP, continuing as an annual amount through the end of the decade, subject to demonstrating continued requirement. This makes the UK AISI an outlier in funding compared to other national AI Safety Institutes globally.
Early Development (2023-2024)
Section titled “Early Development (2023-2024)”2023 activities: The Institute rapidly hired technical staff, recruiting senior alumni from OpenAI, Google DeepMind, and the University of Oxford. Ian Hogarth was appointed as permanent Chair in June 2023. By the end of 2023, the organization had established its organizational structure, built relationships with AI labs for model access, and begun coordination with the US AISI.
2024 milestones:
| Date | Development |
|---|---|
| April 2024 | Published first evaluation results on publicly available frontier models |
| May 2024 | Open-sourced Inspect AI evaluation framework |
| May 2024 | Announced San Francisco office opening |
| May 2024 | AI Seoul Summit: formed International Network of AI Safety Institutes |
| November 2024 | First meeting of International Network in San Francisco |
| December 2024 | Published Frontier AI Trends Report |
Current status (2025): Following the February 2025 renaming to AI Security Institute, the organization now focuses more explicitly on national security threats. Technology Secretary Peter Kyle announced the change at the Munich Security Conference, emphasizing a pivot from concerns about bias and misinformation toward risks such as cyberattacks, fraud, and AI-enabled bioweapon development. The Institute has launched a new criminal misuse team in partnership with the Home Office to investigate AI’s role in serious crimes including child exploitation, cybercrime, and financial fraud.
Core Functions and Activities
Section titled “Core Functions and Activities”1. Frontier Model Evaluations
Section titled “1. Frontier Model Evaluations”The Institute independently assesses advanced AI systems for dangerous capabilities across domains critical to national security and public safety. In their April 2024 evaluation of publicly available frontier models, AISI found several concerning capabilities:
| Evaluation Domain | Key Findings (April 2024) |
|---|---|
| Cybersecurity | Models capable of completing basic to intermediate cybersecurity challenges |
| Chemistry/Biology | Several models demonstrated PhD-equivalent level of knowledge |
| Safeguards | Built-in safeguards vulnerable to even basic “jailbreaks” |
| Autonomous Systems | Models performed better than biology PhD experts on open-ended questions |
Methodology: The Institute has built evaluation expertise by learning from and collaborating with organizations like METR, Apollo Research, and ARC. Their approach includes both pre-deployment and post-deployment testing, with a focus on red-teaming and adversarial evaluation techniques.
Lab cooperation: AISI has established voluntary agreements with OpenAI, Anthropic, and Google DeepMind for pre-release access to models. Labs share safety-relevant information under NDAs, coordinate on safety thresholds, and collaborate on evaluation methodology development.
Inspect AI Framework: In May 2024, AISI open-sourced Inspect AI, a Python framework for building and running reproducible LLM evaluations. The framework has been adopted by major AI labs including Anthropic, DeepMind, and xAI. Inspect AI includes over 100 pre-built evaluations, a web-based visualization tool, and support for tool calling and multi-agent evaluations. Key benchmarks include ChemBench (2,786 chemistry questions), Humanity’s Last Exam (3,000 questions across dozens of subjects), and Cybench (40 professional-level cybersecurity challenges).
2. Research Funding Programs
Section titled “2. Research Funding Programs”AISI manages substantial external research funding, making it one of the largest funders of AI safety research globally. The Institute’s grant programs combine government funding with partnerships from industry and philanthropy.
| Program | Total Funding | Per-Project Range | Focus Areas |
|---|---|---|---|
| The Alignment Project | 15 million GBP | 50K-1M GBP | Alignment research, interdisciplinary collaboration |
| Systemic Safety Grants | 8.5 million GBP | Up to 200K GBP | Societal resilience, healthcare, energy grids, financial markets |
| Challenge Fund | 5 million GBP | Up to 200K GBP | Safeguards, control, alignment, societal resilience |
The Alignment Project: This global fund is supported by an international coalition including the UK AISI, Canadian AI Safety Institute, Schmidt Sciences, Amazon Web Services, Halcyon Futures, Safe AI Fund, UKRI, Anthropic, and ARIA. Recipients can also receive up to 5 million GBP in dedicated AWS cloud computing credits, enabling technical experiments beyond typical academic reach.
Systemic Safety Grants: Launched in partnership with the Engineering and Physical Sciences Research Council (EPSRC) and Innovate UK (part of UKRI), the first phase allocated 4 million GBP to support approximately 20 projects. The program aims to increase societal resilience to widespread AI deployment across critical infrastructure.
3. Technical Safety Research
Section titled “3. Technical Safety Research”AISI conducts both in-house and collaborative research across several priority areas:
Technical research: Evaluation methodologies, interpretability and transparency techniques, robustness and reliability testing, alignment approaches, and measurement/benchmarking standards.
Sociotechnical research: AI governance frameworks, risk assessment methodologies, safety standards and best practices, international coordination mechanisms, and public engagement strategies.
Key outputs include: The Frontier AI Trends Report (December 2024), which provides an empirical assessment of frontier AI capabilities and their trajectory, and regular evaluation reports shared with policymakers and the public.
4. International Coordination
Section titled “4. International Coordination”The UK has positioned itself as a convener for international AI safety efforts, hosting the first AI Safety Summit and co-founding the International Network of AI Safety Institutes.
International Network of AI Safety Institutes: During the AI Seoul Summit in May 2024, international leaders agreed to form this network comprising institutes from the UK, US, Japan, France, Germany, Italy, Singapore, South Korea, Australia, Canada, and the European Union. The first meeting was held in San Francisco in November 2024, with representatives from 10 countries and the European Commission.
US-UK partnership: The UK and US AISIs work as “sister organizations” with close collaboration on methodologies, coordinated evaluations, and joint international engagement. The San Francisco office (opened summer 2024) facilitates this partnership and allows AISI to recruit Bay Area talent and engage directly with major AI labs.
Seoul Statement goals: The international network aims to “accelerate the advancement of the science of AI safety” by promoting complementarity and interoperability between institutes and fostering a common international understanding of AI safety approaches.
5. Standard-Setting and Guidelines
Section titled “5. Standard-Setting and Guidelines”Developing best practices:
Evaluation standards:
- Methodologies for dangerous capability testing
- Thresholds for deployment decisions
- Red-teaming protocols
- Transparency and documentation requirements
- Continuous monitoring approaches
Safety standards:
- Risk management frameworks
- Safety assurance processes
- Incident response and reporting
- Governance and accountability
- Alignment verification
UK and international adoption:
- UK government procurement requirements
- Industry adoption (voluntary initially)
- International harmonization efforts
- Foundation for potential regulation
- Soft power and norm-setting
6. Policy Advice and Government Coordination
Section titled “6. Policy Advice and Government Coordination”Advising UK government:
Policy development:
- Risk assessments for ministers and parliament
- Technical input to AI legislation
- Regulatory framework design
- International negotiation support
- Public communication strategies
Interagency coordination:
- National security agencies (GCHQ, MI5, MI6)
- Regulatory bodies (ICO, CMA, Ofcom)
- Research funders (UKRI)
- Science and technology departments
- Foreign policy (Foreign Office)
Parliamentary engagement:
- Expert testimony to committees
- Briefings for MPs
- Technical translation for policymakers
- Risk communication
- Policy option analysis
Key Personnel and Expertise
Section titled “Key Personnel and Expertise”Ian Hogarth (Chair)
Section titled “Ian Hogarth (Chair)”Background:
- Technology entrepreneur and investor
- AI safety advocate
- Wrote influential Financial Times op-ed on “God-like AI”
- Connections to UK tech sector and government
- Strong safety focus
Leadership:
- Strategic direction for AISI
- External relations and advocacy
- Government and international engagement
- Representing AISI publicly
- Ensuring independence and rigor
Approach:
- Taking AI risks seriously
- Building credible technical organization
- International cooperation
- Balancing innovation with safety
- Evidence-based policymaking
Staffing and Expertise
Section titled “Staffing and Expertise”Hiring from:
- AI safety organizations (METR, Apollo, ARC, Anthropic, etc.)
- UK universities (Oxford, Cambridge, Imperial, etc.)
- Frontier AI labs (researchers concerned about safety)
- Other government departments (GCHQ, etc.)
- International talent
Expertise needed:
- Machine learning and AI systems
- Evaluation and red-teaming
- Cybersecurity and CBRN
- Interpretability and alignment
- Policy and governance
- International relations
Hiring challenges:
- Competing with private sector salaries (London is expensive)
- Government hiring processes
- Security clearances required for some roles
- Need for rapid scaling
- Retention in fast-moving field
Advantages:
- Mission-driven work with government backing
- Influence on policy and practice
- Access to frontier models
- International platform
- Job security and meaningful impact
Relationship with AI Labs
Section titled “Relationship with AI Labs”Cooperation framework:
Voluntary Agreements
Section titled “Voluntary Agreements”Lab commitments:
- Provide pre-release access to models
- Share safety-relevant information
- Participate in evaluations
- Engage on standard-setting
- International coordination
AISI commitments:
- Responsible handling of sensitive information
- Constructive engagement
- Technical rigor
- Timely evaluations
- Security of proprietary data
Labs participating:
- OpenAI
- Anthropic
- Google DeepMind
- Others as they develop frontier models
Information Sharing
Section titled “Information Sharing”What labs share:
- Model access for evaluation (pre-deployment)
- Technical documentation
- Safety research findings
- Risk assessments
- Incident reports (if any)
Confidentiality:
- NDAs and secure handling
- Classified information protocols
- Public reporting (aggregated/redacted)
- National security considerations
- Competitive sensitivity
Mutual Benefits
Section titled “Mutual Benefits”For labs:
- Independent validation of safety claims
- Early warning of risks
- Inform better safety practices
- Government relationships
- Social license and legitimacy
For AISI:
- Access to frontier systems
- Understanding of frontier capabilities
- Influence on deployment decisions
- Technical learning
- Real-world testing
Tensions and Challenges
Section titled “Tensions and Challenges”Potential conflicts:
- Labs want speed, AISI wants thoroughness
- Commercial sensitivity vs. transparency
- National security complications
- Regulatory uncertainty
- International competition
Future evolution:
- Currently voluntary, might become mandatory
- Regulatory framework could formalize
- International coordination might create pressure
- Labs might resist if too burdensome
- Balance between cooperation and oversight
International Leadership and Influence
Section titled “International Leadership and Influence”AI Safety Summits
Section titled “AI Safety Summits”Bletchley Park Summit (November 2023):
- UK convened first international AI Safety Summit
- 28 countries and EU committed to AI safety cooperation
- Bletchley Declaration recognized catastrophic risks
- Foundation for international coordination
- UK positioned as leader
Second Summit (2024):
- South Korea and UK co-hosting
- Building on Bletchley commitments
- Deepening international cooperation
- Expanding participation
- Concrete safety measures
Ongoing convenings:
- Regular international meetings
- Technical working groups
- Research collaborations
- Standard harmonization
- Diplomatic engagement
Setting Global Norms
Section titled “Setting Global Norms”UK influence:
Standard-setting:
- Evaluation methodologies becoming international reference
- Safety frameworks adopted by other countries
- Best practices dissemination
- Training and capacity building
- Soft power through technical leadership
Bridging role:
- Between US and Europe
- Academic and government
- Technical and political
- Innovation and safety
- National and international
Challenges:
- China not participating fully
- Different national approaches
- Verification and enforcement
- Balancing openness and security
- Resource constraints
Impact and Accomplishments
Section titled “Impact and Accomplishments”Quantified Outcomes
Section titled “Quantified Outcomes”| Achievement | Impact | Timeframe |
|---|---|---|
| Bletchley Declaration | 28 countries + EU committed to AI safety cooperation | November 2023 |
| International Network | 10+ countries forming coordinated AI safety ecosystem | May 2024 onwards |
| Inspect AI adoption | Framework used by Anthropic, DeepMind, xAI | 2024-2025 |
| Research funding | 28.5 million GBP+ in external grants launched | 2024-2025 |
| AI Research Resource | 300 million GBP investment (tripled from 100M) | 2023-2024 |
| Staff growth | 30+ technical researchers recruited | 2023-2024 |
On UK AI Policy
Section titled “On UK AI Policy”AISI provides technical expertise directly to Parliament and ministers, informing risk assessments for policy decisions and shaping the framework for UK AI regulation. Key policy inputs include the Online Safety Act AI provisions, AI White Paper consultation responses, regulatory framework development, and national security AI strategy.
On Industry Practices
Section titled “On Industry Practices”AISI’s evaluation methods are becoming industry standard. Major labs (OpenAI, Anthropic, Google DeepMind) have entered voluntary agreements for pre-deployment testing, with Inspect AI being adopted as the evaluation framework of choice. This has normalized pre-deployment safety assessment, increased transparency requirements, and established red-teaming as standard practice.
On International AI Safety
Section titled “On International AI Safety”The UK is now recognized as a global leader in AI safety convening. Concrete outcomes include the Bletchley Declaration (first international agreement on AI risks), the International Network of AI Safety Institutes (10+ member countries), shared evaluation methodologies through Inspect AI, the International Scientific Report on AI Safety (led by Turing Award winner Yoshua Bengio), and foundation for ongoing governance discussions at summits in Seoul and Paris.
Criticisms and Challenges
Section titled “Criticisms and Challenges”The 2025 Rebrand Controversy
Section titled “The 2025 Rebrand Controversy”The February 2025 renaming from “AI Safety Institute” to “AI Security Institute” drew significant criticism from the AI safety community. AI Now Institute noted that the change signals the institute “will no longer focus on AI ethical issues, such as algorithm bias or protecting freedom of speech in AI applications.” Critics worry this represents a narrowing of scope away from broader societal harms toward a purely national security focus.
Defenders argue the rebrand reflects AISI’s actual priorities from the beginning, with Ian Hogarth emphasizing that “this change of name better reflects the team’s essential remit and the priorities that they have always focused on.”
Can Government Keep Pace?
Section titled “Can Government Keep Pace?”AI capabilities advance faster than government institutions can respond. AISI faces structural challenges including bureaucratic constraints on hiring speed, difficulty matching private sector salaries (despite being competitive within civil service), limited compute resources compared to lab budgets, and inevitable evaluation lag behind the frontier.
AISI’s response has been to operate as a “startup inside government” with fast-track hiring, partnerships with academic and private sector organizations, and focus on high-impact evaluation work rather than trying to match lab resources directly.
Regulatory Capture Risk
Section titled “Regulatory Capture Risk”There are concerns that AISI might be captured by industry interests given its dependence on voluntary lab cooperation, the information asymmetry favoring labs, and the revolving door between government and industry. However, mitigations include government independence, diverse expertise sourcing (not only from labs), public interest mandate, parliamentary oversight, and increasing transparency.
International Coordination Challenges
Section titled “International Coordination Challenges”China’s absence from Western AI safety coordination remains a significant limitation. The International Network of AI Safety Institutes includes primarily democratic allies, with no clear pathway to engaging China despite early participation in the Bletchley Declaration. This creates risks of fragmented governance and potential race dynamics.
Resources and Scope
Section titled “Resources and Scope”With an annual budget of approximately 50 million GBP, AISI remains smaller than major lab research budgets and must prioritize carefully. The Institute cannot evaluate everything and depends heavily on partnerships. Maintaining quality and credibility while scaling remains an ongoing challenge.
❓Key Questions
Future Directions
Section titled “Future Directions”Near-Term (1-2 years)
Section titled “Near-Term (1-2 years)”Organizational development:
- Complete initial hiring and team building
- Establish robust evaluation infrastructure
- Deepen lab partnerships
- Expand international collaboration
- Demonstrate technical credibility
Key deliverables:
- Regular frontier model evaluations
- Published research and standards
- Second AI Safety Summit outcomes
- International coordination agreements
- Policy recommendations to UK government
Medium-Term (2-5 years)
Section titled “Medium-Term (2-5 years)”Institutionalization:
- Mature organization with stable funding
- Recognized global leader in AI safety
- Influential in international standards
- Comprehensive evaluation program
- Growing team and capabilities
Possible developments:
- Regulatory authority (if legislation passed)
- Mandatory evaluation requirements
- International verification mechanisms
- Expanded scope (multimodal, robotics, etc.)
- Regional presence beyond London
Long-Term Vision
Section titled “Long-Term Vision”UK as global AI safety hub:
- Leading technical expertise
- Convening power for international cooperation
- Standards and norms shaping global governance
- Preventing catastrophic AI deployments
- Contributing to beneficial AI development
Broader impact:
- Effective international AI governance
- Safe development of transformative AI
- UK playing key role in existential risk reduction
- Model for other countries
- Public trust in AI governance
Comparisons to Other Organizations
Section titled “Comparisons to Other Organizations”vs US AI Safety Institute
Section titled “vs US AI Safety Institute”Similarities:
- Both government AI safety bodies
- Similar missions and approaches
- Close coordination and partnership
- Created around same time (2023)
Differences:
- UK: Smaller budget and country, more nimble
- US: Larger resources, broader scope
- UK: Explicit international hub strategy
- US: Within NIST, different structure
- UK: More unified government approach
- US: More complex federal system
Relationship: Sister organizations, very close collaboration
vs Private Safety Organizations
Section titled “vs Private Safety Organizations”Complementary:
- METR, Apollo, ARC: Private orgs focused on research and evaluation
- UK AISI: Government body with policy authority
- AISI learns from and hires from safety orgs
- Different authorities and mandates
- Collaborative ecosystem
vs Frontier Labs
Section titled “vs Frontier Labs”Different roles:
- Labs: Build AI systems, commercial
- AISI: Evaluate and oversee, government
- Labs: Innovation focus
- AISI: Safety focus
- Cooperative but some tension
- AISI provides independent assessment
Key Developments Timeline
Section titled “Key Developments Timeline”| Date | Event |
|---|---|
| June 2023 | Frontier AI Taskforce launched; Ian Hogarth appointed as Chair |
| November 2023 | AI Safety Summit at Bletchley Park; 28 countries sign Bletchley Declaration |
| November 2023 | UK announces creation of AI Safety Institute |
| April 2024 | First evaluation results published on publicly available frontier models |
| May 2024 | Inspect AI evaluation framework open-sourced |
| May 2024 | San Francisco office announced |
| May 2024 | AI Seoul Summit: International Network of AI Safety Institutes formed |
| May 2024 | International Scientific Report on AI Safety published (led by Yoshua Bengio) |
| November 2024 | First meeting of International Network in San Francisco (10 countries + EU) |
| December 2024 | Frontier AI Trends Report published |
| February 2025 | Renamed to AI Security Institute at Munich Security Conference |
| February 2025 | Criminal misuse team launched with Home Office |
Sources and Resources
Section titled “Sources and Resources”Official Sources
Section titled “Official Sources”- UK AI Security Institute Official Website↗ - Main portal for the Institute
- GOV.UK AI Safety Institute Page↗ - Government organizational page
- Introducing the AI Safety Institute↗ - Official overview document
- Fourth Progress Report↗ - May 2024 progress update
- Our 2025 Year in Review↗ - 2025 annual review
Research Outputs
Section titled “Research Outputs”- Frontier AI Trends Report↗ - December 2024 assessment of frontier AI capabilities
- Inspect AI Framework↗ - Open-source evaluation framework documentation
- Inspect Evals↗ - Collection of 100+ pre-built evaluations
- Early Lessons from Evaluating Frontier AI Systems↗ - Evaluation methodology insights
Funding Programs
Section titled “Funding Programs”- The Alignment Project↗ - 15 million GBP global alignment research fund
- Grants Overview↗ - Information on all AISI grant programs
- Systemic Safety Grants↗ - 8.5 million GBP societal resilience program
International Coordination
Section titled “International Coordination”- The Bletchley Declaration↗ - November 2023 international agreement
- Chair’s Summary of AI Safety Summit 2023↗ - Summit outcomes
- CSIS: The AI Safety Institute International Network↗ - Analysis of international coordination
- Centre for Future Generations: The AI Safety Institute Network↗ - Comparative analysis
News and Analysis
Section titled “News and Analysis”- UK AI Safety Institute Wikipedia↗ - Comprehensive background
- Computer Weekly: San Francisco Office Announcement↗ - May 2024 expansion
- Infosecurity Magazine: AISI Rebrands↗ - February 2025 name change
- AI Now Statement on Transition↗ - Critical perspective on rebrand
Related Pages
Section titled “Related Pages”What links here
- AI Safety Summit (Bletchley Park)historical
- Apollo Researchlab-research
- Conjecturelab-research
- METRlab-research
- ARCorganization
- US AI Safety Instituteorganization
- EU AI Actpolicy
- International AI Safety Summit Seriespolicy
- Executive Order on Safe, Secure, and Trustworthy AIpolicy