US AI Safety Institute
Overview
Section titled “Overview”Quick Facts
Section titled “Quick Facts”| Attribute | Details |
|---|---|
| Established | November 2023 (announced at UK AI Safety Summit) |
| Parent Agency | National Institute of Standards and Technology (NIST), Department of Commerce |
| Director | Elizabeth Kelly (appointed February 2024) |
| Chief Technology Officer | Elham Tabassi |
| Initial Budget | $10 million (March 2024) |
| Consortium Members | 280+ organizations (as of December 2024) |
| 2025 Status | Renamed to Center for AI Standards and Innovation (CAISI) in June 2025 |
The US AI Safety Institute (US AISI) represents the United States government’s primary institutional response to mounting concerns about artificial intelligence risks. Established in 2023 within the National Institute of Standards and Technology (NIST), the institute serves as the federal government’s central hub for AI safety research, evaluation, and standard-setting. Created in the wake of breakthroughs like ChatGPT and growing awareness of potential catastrophic risks from advanced AI systems, US AISI bridges the gap between cutting-edge technical research and government policy oversight.
The institute’s core mission encompasses developing safety standards for AI systems, conducting independent evaluations of frontier models before deployment, coordinating with industry and international partners, and providing technical expertise to inform regulation. Unlike purely academic research organizations or private lab safety teams, US AISI operates with the unique authority and responsibility that comes from government backing, positioning it to influence both industry practices and international AI governance frameworks.
The creation of US AISI marks a fundamental shift in AI safety from a primarily academic and private sector concern to an official government priority with institutional backing and potential regulatory authority. As the first major government entity dedicated specifically to AI safety, its development and effectiveness will significantly influence how democratic societies govern transformative AI technologies.
Founding and Historical Context
Section titled “Founding and Historical Context”The US AI Safety Institute emerged from a perfect storm of technological advancement and policy awakening that characterized 2023. The public release of ChatGPT in November 2022 had crystallized concerns about rapid AI progress that had been building within the research community for years. Simultaneously, international developments including the EU’s AI Act negotiations and the UK’s announcement of its own AI Safety Institute created pressure for the United States to establish its own institutional capacity.
Key Milestones
Section titled “Key Milestones”| Date | Event |
|---|---|
| October 30, 2023 | President Biden signs Executive Order 14110↗ on AI safety |
| November 2023 | VP Harris announces US AISI at UK AI Safety Summit |
| February 8, 2024 | Elizabeth Kelly appointed Director↗; Elham Tabassi appointed CTO |
| February 8, 2024 | AI Safety Institute Consortium (AISIC) launched↗ with 200+ members |
| March 2024 | $10 million budget allocated |
| April 2024 | US-UK bilateral agreement signed for joint safety testing |
| May 2024 | International Network of AI Safety Institutes↗ launched at Seoul Summit |
| August 29, 2024 | Formal agreements signed with Anthropic and OpenAI↗ for model access |
| November 2024 | First joint US-UK evaluation↗ of Claude 3.5 Sonnet published |
| December 2024 | First AISIC plenary meeting↗ at University of Maryland |
| January 2025 | Executive Order 14110 revoked by Trump administration |
| June 3, 2025 | Renamed to Center for AI Standards and Innovation (CAISI)↗ |
The institute was formally announced as part of President Biden’s Executive Order on Safe, Secure, and Trustworthy Artificial Intelligence in October 2023, which represented the most comprehensive US government action on AI to date. The decision to house the institute within NIST reflected strategic thinking about leveraging the agency’s technical credibility, history of successful standard-setting in areas like cybersecurity, and non-partisan scientific reputation. NIST’s placement within the Department of Commerce also signaled the administration’s intent to balance safety concerns with innovation and competitiveness considerations.
The institute’s early development was significantly influenced by the AI safety research community. Key hires included personnel from organizations like METR (formerly ARC Evals), Apollo Research, and other technical safety organizations, bringing established methodologies for evaluating dangerous AI capabilities into government. This brain drain from private safety organizations into government represented both an opportunity to build institutional capacity quickly and a challenge for the broader ecosystem of AI safety research.
By 2024, the institute had begun conducting its first formal evaluations of frontier AI systems, established initial partnerships with major AI laboratories, and started developing comprehensive safety standards. The institute’s rapid initial progress reflected both the urgency of its mission and the advantage of building on existing evaluation methodologies developed by the safety research community.
AI Safety Institute Consortium (AISIC)
Section titled “AI Safety Institute Consortium (AISIC)”In February 2024, the AI Safety Institute Consortium (AISIC)↗ was launched with over 200 members, later growing to 280+ organizations. The consortium brings together diverse stakeholders to develop science-based guidelines and standards for AI measurement and safety.
Notable Consortium Members by Category
Section titled “Notable Consortium Members by Category”| Category | Key Members |
|---|---|
| Frontier AI Labs | OpenAI, Anthropic, Google, Meta, Nvidia, Microsoft |
| Major Tech Companies | Apple, Amazon, IBM, Intel, Salesforce, Palantir, Adobe, Databricks |
| Financial Services | Bank of America, JP Morgan Chase, Citigroup, Wells Fargo, Mastercard |
| Academia | MIT, Stanford, UC Berkeley, CMU, University of Maryland |
| Healthcare/Industry | Boston Scientific, and numerous sector-specific organizations |
| Civil Society | Various nonprofit organizations and research institutes |
The consortium received over 600 letters of interest from organizations seeking to participate. In December 2024, members gathered for the first in-person plenary meeting↗ at the University of Maryland to review progress and plan 2025 priorities.
Organizational Structure and Authority
Section titled “Organizational Structure and Authority”US AISI operates within NIST’s unique institutional position as a non-regulatory federal agency focused on technical standards and scientific research. This placement provides several critical advantages: NIST’s established credibility with industry, its history of successful public-private partnerships in developing technical standards, and its reputation for scientific rigor over political considerations. The institute inherits NIST’s culture of evidence-based decision-making and collaborative standard-setting processes that have proven effective in areas like cybersecurity frameworks.
However, this non-regulatory status also creates important limitations. Unlike agencies such as the FDA or EPA, US AISI cannot directly mandate compliance with its standards or block the deployment of AI systems it deems unsafe. Instead, the institute must rely on a combination of moral authority, industry cooperation, coordination with other agencies that do have regulatory power, and potential future legislative changes to give it enforcement capabilities.
The institute’s authority currently derives from several sources. The October 2023 Executive Order requires companies developing AI systems above certain computational thresholds to report safety test results to the government, providing US AISI with its first formal oversight mechanism. Additionally, the institute’s role in developing standards that may inform future regulation, its participation in government contracting decisions, and its coordination with regulatory agencies like the FTC creates indirect but meaningful influence over industry behavior.
Leadership under Director Elizabeth Kelly has focused on building the institute’s technical credibility and maintaining productive relationships with AI laboratories while establishing clear independence from industry influence. The staffing strategy has prioritized hiring researchers with deep technical expertise in AI safety evaluation, often from the most respected organizations in the field, combined with policy professionals who can translate technical concerns into actionable government policy.
Model Evaluations
Section titled “Model Evaluations”US AISI has conducted several formal model evaluations, with the most significant being the first-ever joint government evaluation with the UK AI Safety Institute.
Completed Model Evaluations
Section titled “Completed Model Evaluations”| Model | Date | Evaluation Type | Key Domains Tested | Notable Findings |
|---|---|---|---|---|
| Claude 3.5 Sonnet (upgraded) | November 2024 | Joint US-UK pre-deployment↗ | Biological, cyber, software development, safeguard efficacy | ”Most comprehensive government-led safety evaluation of an advanced AI model to date” - Elizabeth Kelly |
| OpenAI o1 | December 2024 | Joint US-UK pre-deployment↗ | Biological, cyber, reasoning capabilities | Advanced reasoning model with novel chain-of-thought architecture |
The Claude 3.5 Sonnet evaluation was particularly significant as the inaugural joint evaluation under the April 2024 US-UK bilateral agreement. Testing was conducted by expert engineers, scientists, and subject matter specialists from both institutes, with findings shared with Anthropic before public release.
Core Functions and Capabilities
Section titled “Core Functions and Capabilities”The institute’s evaluation capabilities represent its most direct contribution to AI safety oversight. Building on methodologies developed by organizations like METR and Apollo Research, US AISI conducts comprehensive assessments of frontier AI models before and after deployment, focusing particularly on dangerous capabilities that could pose risks to national security or public safety. These evaluations examine models for concerning capabilities across domains including cybersecurity (ability to find vulnerabilities or assist in cyberattacks), weapons development (particularly chemical, biological, radiological, and nuclear knowledge), autonomous decision-making that could lead to loss of human control, and deceptive or manipulative capabilities that could undermine democratic institutions.
The institute’s standard-setting work builds on NIST’s successful AI Risk Management Framework, which has been voluntarily adopted by hundreds of organizations worldwide. This framework provides a structured approach for organizations to identify, assess, and mitigate AI-related risks throughout the AI lifecycle. US AISI continues to develop more specific technical standards covering areas such as evaluation methodologies, safety testing protocols, documentation requirements for AI systems, and metrics for measuring AI system safety and robustness.
Research collaboration represents another core function, with the institute serving as a bridge between government needs and the broader AI safety research community. Through grant programs, collaborative research agreements, and partnership arrangements, US AISI works to ensure that safety research addresses the most pressing governance challenges while maintaining scientific independence and rigor. The institute also plays a crucial role in international coordination, working closely with counterparts like the UK AI Safety Institute to develop harmonized approaches to AI safety evaluation and governance.
The institute’s policy support function provides technical expertise to lawmakers, regulators, and other government agencies grappling with AI governance challenges. This includes conducting risk assessments to inform regulatory decisions, providing Congressional testimony on technical AI issues, and coordinating across federal agencies to ensure consistent approaches to AI oversight. This technical translation role is critical given the complexity of AI systems and the need for evidence-based policy-making.
Industry Relationships and Cooperation
Section titled “Industry Relationships and Cooperation”On August 29, 2024, US AISI signed landmark Memoranda of Understanding↗ with Anthropic and OpenAI, establishing formal frameworks for pre-deployment model access and collaborative safety research.
Formal Agreements with AI Labs
Section titled “Formal Agreements with AI Labs”| Company | Agreement Date | Key Terms | Company Statement |
|---|---|---|---|
| Anthropic | August 29, 2024 | Pre-deployment access to major new models; collaborative safety research | ”Our collaboration with the U.S. AI Safety Institute leverages their wide expertise to rigorously test our models before widespread deployment.” - Jack Clark, Co-founder |
| OpenAI | August 29, 2024 | Pre-deployment and post-deployment access; joint evaluation methodology development | ”We strongly support the U.S. AI Safety Institute’s mission and look forward to working together to inform safety best practices.” - Jason Kwon, Chief Strategy Officer |
These agreements enable US AISI to receive access to frontier models both before and after public release, conduct collaborative research on capability evaluation and risk mitigation, and provide feedback on potential safety improvements in coordination with the UK AI Safety Institute.
The relationship between US AISI and major AI laboratories represents one of the most important and complex dynamics in contemporary AI governance. The institute has established collaborative relationships with all major US AI developers, including OpenAI, Anthropic, Google DeepMind, and Meta, built on a foundation of shared concern about AI risks combined with mutual recognition of each party’s legitimate interests and constraints.
From the laboratories’ perspective, cooperation with US AISI serves multiple strategic purposes. Voluntary engagement demonstrates good faith efforts at responsible development, potentially influencing future regulatory frameworks to be more favorable or proportionate. Early sharing of safety information allows labs to receive government feedback before public deployment, potentially identifying problems that could generate negative publicity or regulatory backlash. Participation in standard-setting processes gives labs influence over the rules they will eventually need to follow, while collaboration helps build relationships with government officials who may hold regulatory power in the future.
However, this cooperation exists within a framework of underlying tensions. AI laboratories face intense competitive pressure to deploy new capabilities quickly, while thorough safety evaluation takes time and resources. Companies must balance transparency with protection of proprietary information and competitive advantages. There are also legitimate concerns about government access to sensitive technical information, particularly given national security implications and the potential for information to reach competitors or adversaries.
The current cooperative framework relies heavily on voluntary compliance and industry goodwill, creating inherent fragility. Major labs have generally been willing to share information and participate in evaluations, but this cooperation could break down if government demands become too burdensome, if competitive pressures intensify, or if labs conclude that cooperation is not producing favorable regulatory outcomes. The institute’s challenge is maintaining productive relationships while building credible oversight capabilities that serve the public interest even when industry preferences diverge from broader safety considerations.
International Coordination and Leadership
Section titled “International Coordination and Leadership”At the AI Seoul Summit in May 2024↗, the International Network of AI Safety Institutes was formally launched, with the US playing a leading role alongside the UK, EU, Japan, and other partners.
International Network of AI Safety Institutes
Section titled “International Network of AI Safety Institutes”| Member | Institute/Office | Status (as of 2024) |
|---|---|---|
| United States | US AI Safety Institute (NIST) | Founding member; co-host of inaugural convening |
| United Kingdom | UK AI Safety Institute | Founding member; closest bilateral partner |
| European Union | European AI Office | Founding member |
| Japan | AI Safety Institute | Founding member |
| Canada | Canadian AI Safety Institute | Founding member |
| Singapore | AI Safety Institute | Founding member |
| South Korea | Korean AI Safety Institute | Host of Seoul Summit |
| France | AI Safety Office | Founding member; host of Paris 2025 Summit |
| Germany | AI Safety Office | Founding member |
| Australia | AI Safety Office | Founding member |
| Kenya | AI Safety Office | Founding member |
The inaugural convening of the Network↗ was co-hosted by the US Departments of Commerce and State on November 20-21, 2024, in San Francisco, bringing together technical experts to align on priority work areas ahead of the AI Action Summit in Paris (February 2025).
US AISI plays a central role in emerging international efforts to coordinate AI safety governance across democratic nations. The institute works most closely with the UK AI Safety Institute, sharing evaluation methodologies, coordinating on international standards, and jointly developing best practices for government oversight of AI development. This partnership reflects both countries’ recognition that effective AI governance requires international coordination given the global nature of AI development and deployment.
The institute participates in broader multilateral forums including G7 discussions on AI governance, OECD AI policy initiatives, and emerging international standards bodies focused on AI safety. These efforts seek to develop harmonized approaches to AI risk assessment, compatible safety standards across jurisdictions, and coordinated responses to dangerous AI capabilities that could threaten international stability. The goal is creating a coherent international framework that prevents regulatory arbitrage while respecting different countries’ governance preferences and capabilities.
However, international coordination faces significant challenges, particularly the exclusion of China from most safety cooperation efforts. Given China’s major role in AI development and deployment, the absence of Chinese participation in safety coordination creates serious limitations on the effectiveness of international efforts. Additionally, different countries are developing divergent regulatory approaches - the EU’s comprehensive AI Act differs substantially from the US preference for sector-specific regulation, while other countries are still developing their governance frameworks.
The institute also confronts the tension between openness and national security in international cooperation. While effective safety governance benefits from information sharing and coordinated evaluation efforts, sensitive information about AI capabilities and vulnerabilities has national security implications that limit what can be shared even with close allies. US AISI must navigate these constraints while building meaningful international cooperation on shared safety challenges.
Current Challenges and Limitations
Section titled “Current Challenges and Limitations”The institute faces several critical challenges that could limit its effectiveness in overseeing rapidly advancing AI capabilities. The most fundamental challenge is building sufficient technical expertise to meaningfully evaluate frontier AI systems that represent the cutting edge of human technological capability. Government hiring processes are notoriously slow, and the institute competes with AI laboratories that can offer significantly higher compensation for top talent. The rapid pace of AI advancement means that evaluation methodologies and safety standards must continuously evolve, requiring not just initial expertise but ongoing adaptation and learning.
Resource constraints compound the expertise challenge. While the institute’s budget is substantial by government standards, it pales in comparison to the resources available to major AI laboratories. This creates limitations on the computational resources available for testing, the number of models that can be thoroughly evaluated, and the scope of research that can be conducted. The institute must carefully prioritize its efforts while building partnerships and collaborative arrangements that leverage external resources.
The institute’s non-regulatory status creates ongoing uncertainty about its ultimate authority and influence. While voluntary cooperation has been effective so far, there is no guarantee that this will continue as AI capabilities advance and competitive pressures intensify. The institute lacks the direct enforcement authority of agencies like the FDA, meaning it must rely on a combination of moral authority, industry goodwill, and coordination with other agencies to influence behavior. This could prove insufficient if facing determined resistance from AI developers or in crisis situations requiring rapid response.
Perhaps most concerning is the fundamental challenge of keeping pace with AI development while maintaining rigorous evaluation standards. The traditional government approach to safety oversight assumes relatively stable technologies that can be thoroughly studied and regulated through deliberate processes. AI development challenges these assumptions through rapid capability improvement, emergent properties that appear unpredictably, and potential for sudden breakthroughs that could render existing safety measures obsolete. The institute must develop new approaches to governance that can maintain safety and public accountability while adapting to technological change that occurs faster than traditional regulatory timelines.
Safety Implications and Risk Assessment
Section titled “Safety Implications and Risk Assessment”The establishment of US AISI represents both promising progress and concerning limitations in society’s preparation for advanced AI risks. On the promising side, the institute provides the first serious government institutional capacity for evaluating and overseeing AI development in the United States. Having technical experts within government who understand AI capabilities and risks ensures that policy decisions will be informed by scientific evidence rather than speculation or industry lobbying. The institute’s evaluation capabilities could identify dangerous AI capabilities before they are deployed at scale, potentially preventing catastrophic accidents or misuse.
The institute’s standard-setting work creates frameworks for responsible AI development that extend beyond any single company’s internal processes. By establishing government-backed standards for AI safety evaluation, risk assessment, and deployment practices, US AISI can influence industry behavior even without direct regulatory authority. The international coordination efforts help ensure that safety standards are not undermined by regulatory arbitrage, where AI development simply moves to jurisdictions with weaker oversight.
However, the limitations of the current approach raise serious concerns about whether existing institutional arrangements are adequate for the risks posed by advanced AI systems. The institute’s reliance on voluntary cooperation creates uncertainty about its effectiveness if industry incentives shift or if labs conclude that cooperation is not in their interests. The resource and expertise constraints mean that not all AI development can be thoroughly overseen, potentially allowing dangerous capabilities to be deployed without adequate evaluation.
Most fundamentally, the institute’s current capabilities may be insufficient for the speed and scale of AI advancement. If AI development accelerates further, or if breakthrough capabilities emerge suddenly, the institute’s deliberate evaluation processes may be too slow to provide meaningful oversight. The potential for AI systems to exceed human capability in domains relevant to safety evaluation creates the possibility that even well-intentioned oversight could miss critical risks or fail to understand the implications of new capabilities.
2025 Transition: Renaming to CAISI
Section titled “2025 Transition: Renaming to CAISI”On June 3, 2025, Secretary of Commerce Howard Lutnick announced the transformation↗ of the US AI Safety Institute into the Center for AI Standards and Innovation (CAISI). This change followed the Trump administration’s revocation of Executive Order 14110 in January 2025.
Key Changes Under CAISI
Section titled “Key Changes Under CAISI”| Aspect | Under AISI (2023-2025) | Under CAISI (2025-) |
|---|---|---|
| Name | US AI Safety Institute | Center for AI Standards and Innovation |
| Focus | Safety-first approach | Innovation and competitiveness focus |
| Regulatory Approach | Standards development with potential for regulation | Light-touch, voluntary standards |
| National Security | One priority among several | Primary emphasis alongside competitiveness |
| International Role | Safety cooperation | ”Represent American interests” and “guard against burdensome regulation by foreign governments” |
According to Secretary Lutnick, CAISI “will evaluate and enhance US innovation of these rapidly developing commercial AI systems while ensuring they remain secure to our national security standards.” The center remains housed within NIST and continues to:
- Serve as industry’s primary point of contact for AI testing and collaborative research
- Develop guidelines and best practices for AI system security
- Establish voluntary agreements with private sector AI developers
- Lead unclassified evaluations of AI capabilities that may pose national security risks
- Conduct assessments of US and adversary AI systems
The name change notably removes “safety” from the title, signaling a shift in emphasis while the primary evaluation functions continue. The AISIC consortium formed under the previous administration remains active.
Future Trajectory and Evolution
Section titled “Future Trajectory and Evolution”Over the next one to two years, CAISI is likely to focus on consolidating its foundational capabilities while expanding its influence within existing institutional constraints. The institute will continue hiring key personnel, particularly senior researchers with established reputations in AI safety evaluation, and building out its technical infrastructure for conducting comprehensive assessments of frontier models. The development of more detailed safety standards and evaluation methodologies will provide clearer guidance for industry while establishing the institute’s technical credibility.
The institute’s relationship with AI laboratories will likely deepen and become more formal, potentially including regular reporting requirements, standardized evaluation processes, and more systematic information sharing arrangements. International coordination efforts will expand as other countries build their own AI safety institutions and seek to harmonize approaches. The institute may establish more formal partnership agreements with allied nations and begin influencing international standards bodies and multilateral organizations.
In the medium term of two to five years, US AISI may undergo significant expansion in both scope and authority. Congressional legislation could provide the institute with direct regulatory power, potentially requiring pre-deployment approval for AI systems above certain capability thresholds, similar to FDA approval processes for new medications. The institute’s budget and staffing could expand substantially as AI risks become more salient to policymakers and the public, allowing for more comprehensive oversight and evaluation capabilities.
The institute may also expand beyond its current focus on the most advanced AI systems to cover a broader range of AI applications and deployment contexts. This could include oversight of AI systems used in critical infrastructure, AI-enabled weapons systems, and AI applications in sensitive domains like healthcare, education, and criminal justice. Regional offices or international presence could facilitate coordination with allied nations and provide oversight of global AI development.
Critical Uncertainties and Open Questions
Section titled “Critical Uncertainties and Open Questions”Several fundamental uncertainties will determine US AISI’s ultimate effectiveness and influence in AI governance. The most critical question is whether voluntary cooperation from AI laboratories will prove sustainable as competitive pressures intensify and AI capabilities advance. If cooperation breaks down, it remains unclear whether the institute has sufficient alternative sources of authority and influence to maintain effective oversight, or whether this would require new legislative action to grant regulatory powers.
The pace of AI development relative to institutional adaptation represents another crucial uncertainty. If AI capabilities advance faster than the institute’s ability to understand and evaluate them, traditional governance approaches may prove inadequate regardless of institutional design. This could require entirely new approaches to AI oversight that emphasize real-time monitoring, algorithmic governance, or other mechanisms that can operate at the speed of automated systems.
The international dimension creates additional uncertainties about whether coordinated governance is possible without universal participation, particularly from China. If major AI-developing nations pursue fundamentally different approaches to AI governance, it may undermine efforts to create effective international coordination. The potential for AI development to become increasingly geopolitically competitive could make safety cooperation more difficult even among allied nations.
Perhaps most fundamentally, there remain deep uncertainties about the nature and timeline of AI risks themselves. The institute’s current approach assumes that dangerous capabilities will emerge gradually and be detectable through existing evaluation methods. However, if AI development produces sudden breakthroughs, emergent capabilities that appear unpredictably, or forms of intelligence that exceed human understanding, even well-designed oversight institutions may prove inadequate. The ultimate test of US AISI’s approach will be whether it can adapt to scenarios that challenge its foundational assumptions about the governability of advanced AI systems.
❓Key Questions
Sources and Further Reading
Section titled “Sources and Further Reading”Official Sources
Section titled “Official Sources”- US AI Safety Institute at NIST↗
- Elizabeth Kelly appointment announcement↗
- AISIC Consortium launch↗
- Anthropic and OpenAI agreements↗
- Claude 3.5 Sonnet evaluation↗
- International Network launch↗
- CAISI transformation announcement↗
Analysis and Commentary
Section titled “Analysis and Commentary”- CSIS: The US Vision for AI Safety with Elizabeth Kelly↗
- CSIS: AI Safety Institute International Network Recommendations↗
- Wikipedia: AI Safety Institute↗
What links here
- UK AI Safety Instituteorganization
- Biden AI Executive Orderpolicy