The internet is booming about GPT-3, OpenAI’s new AI language model.
GPT-3 is the most powerful language model ever built. This is primarily due to its size: the model has a whopping 175 billion parameters. To put this figure into perspective, its predecessor the GPT-2 model, which was considered to be state-of-the-art and incredibly massive when released last year, had 1.5 billion settings.
After originally published Its GPT-3 research in May, OpenAI gave some members of the public access to the model last week via an API. Over the past few days, sample text generated by GPT-3 has started to circulate widely on social media.
The language capabilities of GPT-3 are mind-boggling. When properly primed by a human, it can write creative fiction; he can generate operation code; he can compose thoughtfully business notes; and much more. Its possible use cases are only limited by our imagination.
Yet there is widespread misunderstanding and hyperbole about the nature and limits of GPT-3’s capabilities. It is important for the tech community to have a clearer view of what GPT-3 can and cannot do.
Basically, GPT-3 is an extremely sophisticated text predictor. A human gives it a piece of text as input, and the model generates its best guess as to what the next piece of text should be. It can then repeat this process – taking the original entry with the newly generated song, treating it as a new entry, and generating a subsequent song – until it reaches a length limit.
How does GPT-3 generate these predictions? It actually ingested all the text available on the Internet. The output it generates is a language it calculates as a statistically plausible response to the input given to it, based on everything humans have posted online.
Surprisingly rich and nuanced information can be extracted from latent patterns in massive data sets, far beyond what the human mind can recognize on its own. This is the basic premise of modern machine learning. After training on a dataset of half a trillion words, GPT-3 is capable of identifying and dazzling riff on the linguistic models it contains.
But GPT-3 has no internal representation of what these words actually mean. He does not have a semantically based model of the world or the subjects he talks about. We can not say understand its inputs and outputs significantly.
Why is this important? Because it means that GPT-3 does not have the ability to reason in an abstract way; it lacks real common sense. Faced with concepts, content or even formulations for which the existing body of text of the Internet has not prepared him, he is helpless.
GPT-3 cannot be cleverly handled by asking simple questions. As an example, consider next exchange with GPT-3, courtesy of Kevin Lacker:
Human: How many eyes does a giraffe have?
GPT-3: A giraffe has two eyes.
Human: How many eyes does my foot have?
GPT-3: Your foot has two eyes.
Human: How many eyes does a spider have?
GPT-3: A spider has eight eyes.
Human: How many eyes does the sun have?
GPT-3: The sun has an eye.
Human: How many eyes does a blade of grass have?
GPT-3: A blade of grass has an eye.
This makes do not mean that GPT-3 is not a useful tool or that it will not underpin many useful applications. This does mean, however, that GPT-3 is unreliable and susceptible to basic mistakes an average human would never make.
A related drawback is that GPT-3 generates its output word by word, based on the immediately surrounding text. The consequence is that he may struggle to maintain a cohesive narrative or deliver a meaningful message over more than a few paragraphs. Unlike humans, who have a persistent mental pattern – a point of view that endures from moment to moment, day to day – GPT-3 has amnesia, often confusedly wandering off after a few sentences.
Like the OpenAI researchers themselves recognized: “GPT-3 samples [can] lose their consistency over sufficiently long passages, contradict each other and sometimes contain sentences or paragraphs without sequitur.
Simply put, the model lacks a comprehensive and long-term sense of meaning and purpose. This will limit its ability to generate useful language output in many contexts.
There is no doubt that GPT-3 is an impressive technical achievement. It has considerably advanced the state of the art in natural language processing. It has an ingenious ability to generate language in all kinds of styles, which will unlock exciting apps for entrepreneurs and DIYers.
Still, a realistic view of the limitations of GPT is important so that we can get the most out of the model. GPT-3 is ultimately a correlative tool. He cannot reason; it does not understand the language it generates. Claims that GPT-3 is sensitive or that he represents “general intelligence” are silly hyperboles that cloud the public discourse around technology.
In a welcome dose of realism, OpenAI CEO Sam Altman makes the same point earlier today on Twitter: “The hype for GPT-3 is way too big … AI is going to change the world, but GPT-3 is just a very early glimpse.”