Do AI Companies Make Good on Voluntary Commitments to the White House?
Summary
Research analyzed 16 AI companies' compliance with White House voluntary AI commitments in 2023, finding wide disparities in performance with an average score of 53% and significant weaknesses in model weight security and third-party reporting.
Review
The study provides a comprehensive examination of how major AI companies have implemented voluntary commitments made to the White House in 2023. By developing a detailed scoring rubric with 30 indicators across eight commitment areas, the researchers systematically evaluated public disclosures from companies to assess their actual implementation practices. The findings reveal substantial heterogeneity in company performance, with scores ranging from 13.3% (Apple) to 83.3% (OpenAI). Notably, Frontier Model Forum members consistently scored higher, and earlier signatories demonstrated better alignment with commitments. The study identified critical weaknesses, particularly in model weight security (average score of 17%) and third-party reporting, highlighting significant gaps between public commitments and actual practices. The research underscores the need for more precise, targeted, and verifiable voluntary commitments in AI governance.
Key Points
- OpenAI scored highest at 83.3%, while Apple scored lowest at 13.3%
- Frontier Model Forum members consistently outperformed other companies
- Model weight security showed systemic poor performance with an average score of 17%
- Voluntary commitments lack clear mechanisms for accountability and verification