From Exponential Growth to Stagnation: What’s Next for Large AI Models?

In Silicon Valley, a belief is slowly growing that could have enormous consequences – the progress of large artificial intelligence (AI) models is slowing down, which are expected to reach human levels in the near future. Since the enthusiastic launch of the conversational robot ChatGPT two years ago, AI proponents believed that its achievements would accelerate exponentially as tech giants facilitated the process with computing power and vast amounts of data to train the models. They thought it depended only on resources: if enough computing power and data were secured, general artificial intelligence (AGI) with equal or greater capabilities than humans would emerge.

The process advanced so quickly that industry leaders, including Elon Musk, called for a moratorium on AI research. However, large tech companies, including Musk’s, did not cease operations but instead spent tens of billions of dollars to avoid falling behind in development. OpenAI, the organization responsible for ChatGPT backed by Microsoft, recently raised $6.6 billion to fund further progress. xAI, Musk’s AI company, is currently raising six billion dollars to purchase 100,000 chips from Nvidia that power the operation of large models. However, there seem to be obstacles on the path to general artificial intelligence.

Industry ‘insiders’ are beginning to acknowledge that large language models (LLMs) cannot grow indefinitely even if they are supplied with more data and computing power. Despite massive investments, research shows signs of stagnation. – The astronomical valuations of companies like OpenAI and Microsoft are largely based on the idea that LLMs will become general artificial intelligence if they continue to expand – said expert and regular AI critic Gary Marcus.

‘No Wall’

One of the fundamental challenges is the limited amount of language data available for training the models. According to Scott Stevenson, head of the AI legal firm Spellbook, reliance on exclusively language data for model expansion must inevitably lead to stagnation. – Some of the labs have been far too focused on just adding more language data to (the model), thinking it will simply continue to get smarter – explained Stevenson.

Sasha Luccioni, a researcher and head of the AI department at the development company Hugging Face, argues that the stagnation was predictable given the companies’ focus on size rather than purpose. – The pursuit of AGI has always been unrealistic, and the approach of ‘bigger is better’ when it comes to artificial intelligence had to reach a limit over time, and I think that is what we are seeing here now – she told AFP.

The industry, however, disputes these interpretations and maintains that progress towards human-level artificial intelligence is unpredictable. – There is no wall – wrote OpenAI’s head Sam Altman on Thursday on the social platform X, but he did not explain what he meant by that. Dario Amodei, head of the company Anthropic, which develops the conversational robot Claude in partnership with Amazon, insists: – If you look at the growth rate of these capabilities, you can really think we will get there by 2026 or 2027 – he emphasized.

Time to Reflect

Nevertheless, OpenAI has postponed the release of the expected successor to the GPT-4 model, which powers ChatGPT, because the increase in its capabilities did not meet expectations, according to sources cited by The Information. The company is now focusing on utilizing its existing capabilities more effectively. The change in strategy is reflected in their recent model o1, designed to provide more accurate answers based on improved reasoning rather than increased data.

Stevenson said that the change in direction at OpenAI to teach its models to spend more time thinking than responding has led to ‘radical improvements’. He compared the emergence of artificial intelligence to the discovery of fire. Instead of adding more fuel in the form of data and computing power, it is time to leverage progress for specific tasks. Stanford University professor Walter De Brouwer compared advanced LLMs to students transitioning from high school to college. – The child of artificial intelligence was a conversational robot that improvised a lot – he noted, adding that it was prone to mistakes. – Now the human approach is arriving, to think before acting – he added.