Key people leave Altman’s OpenAI over security concerns

A few days ago, OpenAI announced exciting news about its new product. They triumphantly declared that their ChatGPT can now converse like a human and has a cheerful and somewhat charming female voice. More specifically, it sounded like Scarlett Johansson, which led the actress to sue them immediately, prompting a quick removal of the feature. However, while the new ChatGPT speaks, it seems that this is not the case for the employees who have recently been leaving Altman’s OpenAI.

First, OpenAI’s co-founder and chief scientist Ilya Sutskever, who led the super alignment team, left the company, followed by the co-lead of the same team Jan Leike.

But that was not the end, as another team member, Gretchen Krueger, also left OpenAI, stating that she shares the concerns of her aforementioned colleagues. She added that more needs to be done to improve ‘decision-making processes; accountability; transparency; documentation; policy implementation; the care with which we use our own technology; and mitigating impacts on inequality, rights, and the environment.’

No surprise

The resignations were reportedly not a complete surprise. Sutskever was involved in a management revolt that led to Altman’s temporary dismissal last year, before the CEO quickly returned to his position. Sutskever publicly regretted his actions and supported Altman’s return, but has since been largely absent from the company, even as other members of OpenAI’s policy, alignment, and security teams have left.

But what really fueled speculation was the silence of the former employees. Sutskever issued a rather typical resignation message, stating that he is ‘confident that OpenAI will build a generative artificial intelligence (AGI) that is both safe and useful, and that he is excited about what lies ahead.’

Leike … did not. His resignation message was simply ‘I resigned‘. After several days of fervent speculation, he expanded on this, explaining that he is concerned that OpenAI has strayed from a safety-oriented culture.

Immediately, questions arose: Were they pushed out? Is this a delayed consequence of Altman’s brief dismissal last autumn? Are the resignations a protest against some secret and dangerous new OpenAI project? Speculation met a void as no one who once worked at OpenAI spoke up.

Fierce NDA

It turns out there is a very clear reason for this. One insider stated that the contracts are so restrictive and contain confidentiality and non-disparagement clauses that former OpenAI employees are subject to. They are prohibited for life from criticizing their former employer, and even acknowledging that the NDA exists is a violation.

If a departing employee refuses to sign the document or violates it, they can lose all the capital they earned during their time at the company, which is likely worth millions of dollars. One former employee, Daniel Kokotajlo, who announced that he left OpenAI ‘due to a loss of trust that it would behave responsibly in the age of AGI,’ publicly confirmed that he had to forfeit a huge amount of money to resign without signing the document.

While confidentiality agreements are not uncommon in the highly competitive Silicon Valley, exposing already earned employee capital to risk due to refusal or violation is. For workers in startups like OpenAI, capital is a vital form of compensation, one that can overshadow their salary. Threatening that money, which could potentially change their lives, is a very effective way to keep former employees silent.

Ultimately, Altman acknowledged in a tweet that there was a provision in the company’s severance documents regarding ‘potential capital forfeiture’ for departing employees, but he stated that the company is in the process of changing that clause in the contracts.

Lack of transparency

All of this is very ironic for a company that initially marketed itself as OpenAI, a committed company to building powerful systems in a transparent and responsible manner.

OpenAI long ago abandoned the idea of open-sourcing its models, citing security concerns. But now it has fired the oldest and most respected members of its security team, which should raise skepticism about whether security is truly the reason why OpenAI has become so closed.

OpenAI has long occupied an unusual position in technological and political circles. Their releases, from DALL-E to ChatGPT, are quite good, but in themselves hardly attract the almost religious fervor with which the company is often discussed.

What sets OpenAI apart is the ambition of its mission: ‘to ensure that general artificial intelligence – AI systems that are generally smarter than humans – benefits all of humanity.’ Many employees believe that this goal is within reach and that with perhaps another decade (or even less) and billions of dollars, the company will succeed in developing AI systems that render most human labor obsolete. Which is, as the company has long stated, as risky as it is exciting.

– Superintelligence will be the most influential technology humanity has ever invented and could help us solve many of the world’s most pressing problems – states the hiring page for Leike’s and Sutskever’s team at OpenAI. ‘But the immense power of superintelligence can also be very dangerous and could lead to the disenfranchisement of humanity or even human extinction. While superintelligence may seem far off now, we believe it could arrive this decade – they write.

Of course, if artificial superintelligence is possible in our lives (and experts are divided), it would have enormous implications for humanity. OpenAI has historically positioned itself as a responsible actor trying to transcend mere commercial incentives and bring generative artificial intelligence for the benefit of all. And they have said they are willing to do so even if it requires slowing down development, passing up profit opportunities, or allowing external oversight.

OpenAI’s unique corporate structure, a capped-profit company ultimately controlled by a non-profit organization, was supposed to enhance accountability. But something has broken, as evidenced by the departures from the company of two technologists with a long history of commitment to safety and a genuine willingness to ask OpenAI to change course if necessary. Sutskever and Leike likely did just that, but it backfired, and they left the company.

Their departure does not signal a change in OpenAI’s mission to build general artificial intelligence – that remains the goal. However, it almost certainly signals a change in OpenAI’s interest in safety.