Anthropic, OpenAI talk safety. Their headcounts don’t

By Parmy Olson

ARTIFICIAL INTELLIGENCE (AI) presents a transformative moment for society, but it appears that the number of people focused on making sure it’s safe might fit on a single transatlantic flight.

Perhaps that shouldn’t surprise given the global arms race that has propelled generative AI companies to stratospheric valuations, but it should cause some alarm. The technology makes errors, is largely untested in the wild and has shown toxic side effects on mental health. Yet a rough estimate of how these companies are staffed suggests a disturbing imbalance, with investment into safety-oriented roles looking like a rounding error compared with the money going into making their systems more powerful.

There are 373 people identified as working full time on making artificial intelligence systems safe and trustworthy at OpenAI, Alphabet Inc.’s Google DeepMind, Anthropic, and Elon Musk’s xAI, a fraction of the more than 11,000 employees estimated to work for these four major AI labs. The estimates comes from Glass.ai, a London-based business intelligence firm. When the labs declined to provide stats on their personnel, I asked Glass.ai to investigate — which it did by crawling LinkedIn profiles, company websites, news articles, and other online sources to identify staff doing safety-related work. That includes tasks like making sure AI tools are aligned with human values, ensuring they’re secure, and that they won’t send users .

To give tech companies the benefit of the doubt, Glass.ai cast a wide net for anyone who might work on making AI safe for human use, including people with generic-sounding titles like “member of technical staff,” bug bounty participants, and safety “fellows” on temporary contracts. The analysis likely misses some relevant staff for each firm, as scraping public data provides only a rough approximation. But it beats the information one can glean directly from AI labs on such figures, which amounts to nothing.

OpenAI, Google DeepMind, and Anthropic pushed back on the estimates as being too low, but again declined to provide alternative figures. Anthropic said that safety was embedded across all its key teams, “not siloed into one function,” and that the estimate of 119 safety staff at the company “materially understates the number of people whose day job is reducing risks from advanced AI.” OpenAI said it had large and growing teams of people working on safety, while Google DeepMind said that safety had always been a top priority. “We have an extensive team contributing to it across the company,” a spokeswoman added. xAI didn’t respond to e-mails seeking comment.

AI firms have about their work. They typically refuse to share details about how their powerful models are developed and what data they are trained on. And a study from Stanford University late last year suggests they . That’s made it harder for researchers or academics to scrutinize the tech and its impact from the outside.

But the companies have also made grand pronouncements about human welfare, often sparked by efforts to claim the moral high ground over rivals. Elon Musk and Sam Altman co-founded OpenAI in 2015 partly as a response to Google’s purchase of DeepMind, saying the latter represented dangerous corporate control of AI, while they would build AI to “benefit humanity.” Dario Amodei split from OpenAI in 2021 to co-found Anthropic as a more “safety-focused” AI lab, asserting that Altman’s venture had become too commercial.

OpenAI made a grand overture toward responsible AI in 2023 when it announced its “Superalignment team” led by Ilya Sutskever and Jan Leike, with a promise to use 20% of its computing power on safety research. Within a year, the team was dissolved, Sutskever and Leike left, and OpenAI “safely” from its mission statement.

Anthropic, meanwhile, has been happy to participate in warfare. Its over domestic surveillance or fully autonomous weapons didn’t stop it from actively seeking and winning Pentagon contracts worth up to $200 million.

However much tech firms brand themselves as safety-focused — to the point of in the case of Anthropic — the real measure comes from where they put their money. Anthropic’s all-important “Constitution,” an 84-page set of moral guidelines for its AI system Claude that tells the model to respect people’s rights and avoid helping them hurt others, was primarily authored by one person.*    

It’s not unusual for humanitarian efforts by tech firms to court and receive outsized public attention. DeepMind’s work on protein folding, which earned co-founder and Chief Executive Officer Demis Hassabis the 2024 Nobel Prize for chemistry and reams of positive press, was largely carried out by a core team of roughly a dozen people, out of workforce of about 4,000.

Ex-staff say the responsibility rhetoric rings hollow, reporting that they’ve rarely seen models be held back over safety concerns or complaining of a reckless culture. After she left OpenAI earlier this year, research scientist Zoe Hitzig that the company was optimizing its models for engagement over user welfare, just as Facebook had done. Mrinank Sharma, who was head of Anthropic’s Safeguards Research Team, also quit in February, writing in a farewell note that he’d “repeatedly seen how hard it is to truly let our values govern our actions,” amid constant “pressures to set aside what matters most.”

No other transformative technology, from nuclear energy to aviation to pharmaceuticals, has been deployed at such speed and with so little insight into what looks like modest safety infrastructure. Those older industries built their safety regimes over decades, often after catastrophic failures. Now we’re expected to trust a few hundred people to oversee AI systems that their own creators predict could or , and with no public accountability on metrics.

The very fact that this column had to be based on crawling LinkedIn and the web, rather than on numbers provided directly by the AI labs, suggests safety might not be the priority they claim it is.

BLOOMBERG OPINION

*Anthropic staffer Amanda Askell told me in an interview that she was the primary author of Claude’s Constitution, with significant input from both internal and external sources including philosophers, ethicists, and theologians.