Description du poste
The aim of this post is to start a discussion about the idea of a shared cluster used by the AI Safety community; we outline the practical problems we’ve seen and some possible solutions, rather than a finished design.
What, Who, Why, HowBeing around several AI safety fellowships and hubs, we’ve noticed a common pattern: we have a generous amount of money for compute; of course we need it to run inference, train models, and extract activations from models of a reasonable size (reasonable enough to exhibit the behaviors we want to study). We usually end up renting GPUs from providers like RunPod or Vast.ai, without much systematic comparison, optimization, or management of the resources. Usually the engineering and research capabilities in these teams are very high, but rarely there is someone expert in cloud infrastructure or ML operations. We figure things out as we go. In practice, this means spending the first weeks of the project setting up environments, downloading model weights (that very likely another team downloaded the week before), and finding out too late that an instance was left running over the weekend, or that a cheaper GPU would have been enough. In our experience it takes a few weeks before finding the right match between compute and actual needs. When the project ends, the machines are released and the next cohort starts again from zero. We have the feeling that, just for our last fellowship’s cohort, tens of thousands of dollars could have been spent better.
Now imagine this happening across dozens of teams and fellowships, working on similar problems, with similar requirements and a need for many of the same tools. How much compute spending could be allocated better? How much infrastructure work are we repeating? Money is not the only reason: even for teams where funding is not the constraint, time is; a fellowship typically lasts a few months, and every week lost on the infrastructure is a week not spent on the research question. This seems esp