Epoch AI training costs
Summary
A comprehensive study examining the dollar cost of training machine learning systems shows training costs have been increasing by around 0.5 orders of magnitude annually, with significant uncertainties and variations between different types of systems.
Review
This research provides a critical examination of the economic trends in AI training, focusing on how the dollar cost of training machine learning systems has evolved between 2009 and 2022. By analyzing a dataset of 124 machine learning systems, the study estimates that training costs have grown by approximately 0.49 orders of magnitude per year, with a 90% confidence interval ranging from 0.37 to 0.56. This growth rate is notably slower than the concurrent growth in computational capabilities, suggesting potential constraints or strategic choices in AI development.
The methodology employs two primary estimation approaches: one using an overall GPU price-performance trend and another using the specific hardware prices of the GPUs used in training. The research highlights significant uncertainties in cost estimation, including variability in hardware prices, utilization rates, and the specific economic contexts of different AI projects. Importantly, the study finds that large-scale systems show a slower growth rate of about 0.2 orders of magnitude per year, indicating potential economic or technological limitations in scaling AI training infrastructure.
Key Points
- Training costs for AI systems have grown by approximately 0.5 orders of magnitude per year from 2009-2022
- Large-scale AI systems show a slower cost growth rate of about 0.2 orders of magnitude per year
- Significant uncertainties exist in cost estimation methods and underlying assumptions