SOURCE-LINKED INTELLIGENCE
Understanding and Fixing Bottlenecks in Optimization for Modern Machine Learning
Understanding and Fixing Bottlenecks in Optimization for Modern Machine Learning Modern machine learning models have been successfully deployed across fields, from scientific studies to tech- nological developments in industry, but their development remains poorly understood. The training of a large language model such as GPT-3 is estimated to cost $4.6M, and public attempts to replicate the training process alone required teams of engineers to rotating on-call for months, monitoring various statistics and constantly tweaking the training procedure when it broke. Existing theoretical frameworks offer limited insights into this process, as they do not capture the main difficulties that arise in practice when training neural networks, leaving practitioners to rely on error-prone heuristics and expensive trial-and-error. This leads not only to a large devel- opment cost dominated by wasted resources, but also limits the possible impacts of machine learn
Read original source ↗ Open in workspace
- recordType
- award
- status
- SIGNED
- region
- EU
- value
- 226420.56
- unit
- EUR
Evidence & attribution
European Commission, CORDIS Horizon Europe project dataset. Metadata adapted.
License: CORDIS reuse policy
First collected: 2026-09-20T03:21:21.440Z. This is not the publication date.