Description du poste
[Crossposted from my Substack.] An update to Ege Erdil’s “What went into training DeepSeek-R1?”
TL;DR• I estimate that DeepSeek-R1-Zero's RL training used ~2e22 (1.5e22-3e22) FLOPs, 30x less than Ege Erdil's estimate of 6.1e23 in the Epoch AI newsletter, mainly due to a batch size correction in the estimate.• I derive a more precise formula for RL compute, accounting for long-context attention costs.• DeepSeek reports that the training run cost ~$200k of GPU time, only 5x less than the previous estimate of $1M.• RL ran at a significantly lower MFU than pre-training (~3% vs 16% respectively). If future estimates don't account for this, FLOP counts during RL may understate the actual costs of training.• I conclude by discussing the implications for RL compute forecasts and thresholds, and outline a research agenda to better track RL compute and investigate scaling laws.
OverviewOn January 20th, 2025, the Chinese AI lab DeepSeek released their flagship open-weight reasoning model, DeepSeek-R1. Its performance was comparable to OpenAI’s o1 — a groundbreaking reasoning model announced 4 months prior. As a result, DeepSeek attracted widespread media attention and discussion around the gap between open and closed-weight models and how much its training cost.
A key enabler for DeepSeek-R1’s performance over its base DeepSeek-V3 was the additional resources spent post-training the model with Reinforcement Learning (from Verifiable Rewards), RL(VR). In this stage, the model learns to reason by completing tasks and updating its weights based on which tasks it successfully completed or not.
Shortly after release, a post by Ege Erdil on Epoch AI’s newsletter attempted to estimate the compute used in training DeepSeek-R1. It calculates that pre-training used 3e24 FLOPs at a cost of $5.3 million, while RL required 6.1e23 FLOPs[1]at a cost of $1 million.
However, Erdil’s post borrows a hyperparameter from the earlier DeepSeekMath paper — batch size, which it noted as a large sourc