This is an audio long read from nature. In this episode, the AI revolution is running out of data.

What can researchers do? Written by Nicola Jones and read by me, Benjamin Thompson.

The internet is a vast ocean of human knowledge, but it isn't infinite, and artificial intelligence

researchers have nearly sucked it dry.

The past decade of explosive improvement in AI has been driven in large part by making neural

networks bigger and training them on ever more data.

This scaling has proved surprisingly effective at making large language models, or LLMs, such

as those that power the chatbot chat GPT, both more capable of replicating conversational

language and of developing emergent properties,

such as reasoning.

But some specialists say that we are now approaching the limits of scaling.

That's in part because of the ballooning energy requirements for computing.

But it's also because LLM developers are running out of the conventional

data sets used to train their models. A prominent study made headlines last year by putting

a number on this problem. Researchers at Epoch AI, a virtual research institute, projected that, by around 2028, the typical size of a

data set used to train an AI model will reach the same size as the total estimated stock

of public online text. In other words, AI is likely to run out of training data in about

three years' time. At the same time,

data owners, such as newspaper publishers, are starting to crack down on how their content can be

used, tightening access even more. That's causing a crisis in the size of the Data Commons,

says Shane Longbray, an AI researcher at the Massachusetts Institute of Technology in Cambridge, who leads the Data Providence Initiative,

a grassroots organization that conducts audits of AI datasets.

...