Description du poste
When Claude Mythos was announced, many people said that we had about six months before open-weights AIs could hack everything, using advanced exploitation and vulnerability-finding capabilities. This timeline for capabilities was correct, and open-weights models have about the same scary capabilities as Mythos did. However, things are not as bad as "everything connected to the internet is probably hacked", and even PHP2Shell seems to not have had especially bad consequences. I think we remain relatively unhacked because exploitation is still difficult and annoying and there's general "inertia", but the situation is likely not sustainable.
Glasswing and similar projects were meant to patch important software, but left large gaps. OpenAI found 948 local privilege escalation exploits in a Linux environment recently. This does not suggest a mature, secured software environment.
In security, doubling down on the same process/design (identifying vulnerabilities and submitting them to maintainers to fix) when it demonstrably won't achieve good coverage in practice is frowned upon. Anthropic's process is enormously lossy, maintainers are struggling to process the reports they do get, and lots of software is probably not being covered or has organizational bottlenecks preventing it from being fixed at any useful rate. Also, I don't think that cyberattacks in practice usually involve flashy zero-days or extremely clever-sounding vulnerabilities, rather than boring misconfigurations, human error or unanticipated interactions between minor problems.
We should instead design systems which will hold up better even with many broken parts. Based on admittedly brief web searches, there is not a detailed plan for this, and most AI-driven effort is focused on finding vulnerabilities, or more speculative, technically shinier, longer-term formal verification. Here are some ideas I think are worth trying, which would benefit from cheap software through AI labour:
Practical hypervisor