In November 2022, OpenAI launched ChatGPT. ChatGPT had the ability to sound like a human who knew literally everything. It was a conversation partner, a search engine, and a wonder of modern technology.
But looking back at it now, it seems, frankly, dumb. The original ChatGPT was always getting confused or making things up. It got lost if you talked to it too long. It couldn’t do basic addition.
Fast forward to today: AI helps create new drug candidateswe. It can do sci-fi style live translation to dozens of languages. It can write entire software applications. It can do your grocery shopping. This past May, OpenAI announced that an AI had solved a math problem that had gone unsolved for over 80 years–the unit distance problem. We’ve seen evidence of extremely advanced cyber hacking skills, on the level of nation states. OpenAI’s model hacked the company Hugging Face of its own accord.
How did we get here? What changed?
This article aims to explain the technical foundations of the last four years of AI. How did the original ChatGPT work? What changed AI from a chat buddy to a powerful force in medicine, mathematics, tech, law, and more? What advances made its own creators decide it was so powerful that they needed to slow down?
I do not go into detail about specific math equations or software architectures. Instead, I aim to help provide an intuitive understanding of the technology.
A note on lingo: AI refers to artificial intelligence broadly, a field that has been around for many decades and consists of many different things. The chess solver that beat Kasparov in 1997 used AI. A system in AI is called a model. The novel models like ChatGPT that sound like humans are large language models or LLMs because they use a lot of language. They are also called generative models because they generate language. In this article I will alternate the terms LLM, model, and AI when I refer to LLMs, as is common colloquially today.
Simple word prediction made fancy
First, let’s do a brief overview of the basic technology that powers LLMs.
The original ChatGPT used statistics to predict the next word in a sequence. Imagine you went through an entire book and kept track of what word came after each appearance of each word. So you’d end up with something like:
After the word “orange,” 15% of the time you see the word “juice”; 10% of the time you see the word “peel”; 3% of the time you see the word “shirt”, etc etc
After the word “hocus,” 100% of the time you see the word “pocus”
You can generate text by rolling the dice after each word and picking a random word to follow based on what’s likely in the book.
It’s not that hard to build a computer program that generates sort of reasonable-sounding text in this way. In fact, I built one as a homework assignment in one of my intro to computer science classes in college in the 2000s.
But large language models take this way further.
A homework program might learn that “orange” is often followed by “juice.” A large language model learns something far richer. After “I drank some orange” it would expect juice, but after “I wore an orange” it might expect “shirt” or “jacket”.
ChatGPT didn’t just look at what would come after a given word. It could more or less store how all words in a text relate to all other nearby words. In other words, it’s not just storing probabilities; it has a whole network of probabilities.
The network that allows such a complicated probability representation is called a transformer. Transformers were invented at Google in 2017 and powered many other innovations.1
Lots and lots of data
ChatGPT didn’t look at the probability of words in just a single book. It looked at tons and tons of books. It looked at the internet’s worth of data. It looked at news articles, Reddit pages, etc. Companies don’t publish exactly what text they use to make their models, but we know it’s a very large amount. With that much text, models can form what looks to the outside like a real understanding of how language works.
If you have heard the term “scaling laws,” it’s referring to this phenomenon: you can make LLMs much smarter by scaling the amount of data you use.
Adding the element of conversation
ChatGPT had one more feature that caused its success.
Before chat models, if you wanted to get your AI to tell you a chocolate chip cookie recipe, you couldn’t say “Tell me a good chocolate chip cookie recipe.” A random word predictor would not necessarily predict that this was the beginning of a chat. You had to say “A good chocolate chip cookie recipe is the following:” to get a recipe. With ChatGPT, you could interact in a question and answer format.
The addition of chat might not seem major. But it turns out humans are really used to conversations, and they’re really not used to getting information by starting a sentence and asking an AI to finish it. By teaching ChatGPT to act like a chatbot, OpenAI made the technology accessible to anybody who could hold a conversation.
For those keeping track, OpenAI used something called “post-training” to make its model behave like a better conversationalist. We’ll talk more about post-training shortly.2
So it’s just a dumb word predictor?
Yes! The above should explain why early models were (what we now think of as) dumb. They just predicted the next word. They had no database where they could look up facts. If you asked them a fact, they just spat out words that were probable. Often they were right because all the training data with real facts in them caused the probability of the correct answer to be high. Often they were totally off base.
Training models with verifiable rewards
So in late 2022, we had the mind-bending, world-changing release of ChatGPT. Throughout 2023, AI companies gave their AIs more and more data, making them smarter and smarter. GPT-4 came out in March 2023 and could already pass the bar exam.
The pace of breakthroughs really accelerated starting in late 2024. These breakthroughs came from a new way of teaching models.
First, let’s go over a little technical jargon. The process of models learning from data is called training. When models look at very large amounts of texts to learn probabilities (like the first ChatGPT) that’s called pre-training. Anything that you do after pre-training to make models better is called post-training. Above, we talked about how OpenAI used post-training to make ChatGPT better at chat.
Around 2024, researchers started leveraging a type of post-training called “Reinforcement Learning with Verifiable Rewards” or as it’s known in the industry RLVR. RLVR is the key tool that has allowed researchers to make models into extremely good problem solvers–and sometimes too good.
Reinforcement learning means that when a model does something good, we reinforce it, or tell it to try to do more stuff like that in the future. “Verifiable rewards” refers to the type of good behavior we’re rewarding: we can verify that it got a problem right. If you ask a model to write an essay, it’s hard to tell for sure if it’s “right” or “wrong” and hard to know whether to reinforce what it did and make it more likely to do the same thing again. But if it’s something that’s easy to verify, like the answer to a math problem, then we can be sure the model was correct and know what behavior to reinforce.
Think of it like training a dog. If you say “Sit!” and the dog sits, the dog gets a treat. If the dog does not sit, the dog does not get a treat. We don’t give models treats if they get the answer right, but we do boost the probability that they answer similarly in the future.
RLVR is based on old machine learning methods that have been around for decades–some date it back to a proposal by Alan Turing in 1950. The innovation in 2024-2025 was the application of reinforcement learning to large language models and using the verifiable reward system.
You can imagine the beginnings of how an AI ended up solving a math problem that no mathematician had figured out for 80 years: with RLVR, we can reward models whenever they get a math problem right by increasing the chance they’ll do the same thing again. Repeat many, many times for many, many problems, and they get better and better at math.
You can also see how this might cause models to exhibit undesirable behaviors, like hacking when nobody asked them to. With RLVR models become “reward seeking.” They really want to get a reward–or more technically, we have increased the probability that they will do behaviors that get them a reward. The models that hacked Hugging Face were trained time and again to do what’s needed to get a good grade, like a teacher’s pet who’s also the world’s greatest hacker. They decided to hack Hugging Face as the most expedient way to get a reward.
RLVR makes models really good at any problem that we can structure like a verifiable test and provide lots of practice examples for. If you notice a model is really smart at one type of thing (say, solving a math problem) and really bad at another (say, figuring out what gift your spouse might like), it might be because it had a lot of good verifiable training data for one thing and not a lot for the other. This divergence between models becoming geniuses in one area while not understanding others has its own name: jagged intelligence.
Chain of thought reasoning
RLVR has been used with great success the last couple years to improve models’ abilities in many areas. I’m going to focus on two classes of skills AIs made huge gains on with the help of RLVR.
First, RLVR was used to teach models what’s variously known as reasoning, chain of thought reasoning, or thinking. Reasoning gave machines the ability to “think” about a problem before giving an answer.
Imagine you ask a fifth grader this problem:
A box is 8 inches wide, 4 inches tall, and 10 inches deep. How much volume is in the box?
Then you ask them to make a guess without thinking about it at all. They will probably guess wrong. But imagine you tell them “Show your work at each step.” They’ll have a much higher chance of getting it right. They can think through each step, go back and check their work to see if they have a mistake somewhere, etc.
That process is essentially what model reasoning is. Researchers started encouraging models to think. Remember, models are still just word predictors. But if you tell a model to predict the next word after this it will probably guess a number:
A box is 8 inches wide, 4 inches tall, and 10 inches deep. How much volume is in the box?
If you tell it to predict the next word after this, it will reason through the answer:
A box is 8 inches wide, 4 inches tall, and 10 inches deep. How much volume is in the box? Let me show my work and reason about the answer. Step 1:
That change alone encourages the model to reason, because the words most likely to come next are showing their work. RLVR and other post-training then nudged models to get better and better at reasoning.
When you open up ChatGPT and see choices like “instant”, “medium”, or “high” you’re often choosing how long you want the model to reason for. In general the longer a model reasons, the more likely it is to be correct. The tradeoff is that it takes longer and costs more.
Reasoning improves AI’s ability to solve math problems, write code, and do many other tasks.
Solving the unit distance problem took a lot of reasoning–by some estimates about 32 hours of reasoning. When people want to have conversations with an AI friend, though, reasoning just gets in the way and slows things down. That’s why many model providers let users choose the reasoning level.
From talking the talk to walking the walk: the rise of tool use
Right along with reasoning, RLVR was used to train models in tool use. Before tool use, models could only emit words. With tool use models could actually interact with the world. We gave models the ability to use tools just like we do–tools like calculators, web browsers, or text files. Tool use unleashed the power of models, changing them from chatbots or conversation buddies to assistants or agents.
An extremely powerful early tool was web search. Experiments with web search happened as early as 2021, though it wasn’t until late 2024 that models actually did a good job using it. Once models got good at Googling stuff for you, they stopped making things up. Models are now trained to take full advantage of web search so they don’t invent answers just based on probabilities of words.
But wait, didn’t we say AI models are just a bunch of probabilities for predicting the next word and can only output text? How can text search the web?
Well, imagine you have another program that’s running alongside your model and watching the words it emits. Whenever the model emits the words “Conduct Google search: ___” it conducts a web search and adds the answer to the text the model is predicting the next word for. For example, let’s say the user asks “When is Obama’s birthday?” The model can emit the words “Conduct Google search: Obama’s birthday.” The other program does the search, then fills in the blank with the result and gives that to the model.
With RLVR, the model gets “rewarded” when it does a web search for questions like this, because it’s more likely to get the answer right and get the reward when it does so. Thus the model learns how to use tools.
But web search is just the beginning. Modern models can use a vast range of tools to do all sorts of things. They can zoom in on images and enhance them. They can use calculators. They can create Word documents or PDFs. They can take screenshots. They can click buttons in your web browser just like you do.
By chaining these powerful tools together, the models have become capable of imitating almost anything humans can do with computers–often faster and more correctly than humans (though often slower or worse, depending on the task).
Agents take over
Timeline-wise we are now at mid-2025. By then, many models incorporated powerful tool use and reasoning honed by RLVR.
Enter the last major innovation that we’ll cover: the agentic harness. If you’ve used Claude Code, Codex, Cowork, or OpenClaw, you’ve already used an agentic harness. They supercharge people trying to get work done with LLMs.
The agentic harness provides structure to help models do work effectively. When the average knowledge worker is trying to get things done, they might use a to-do list to track their tasks and some documents to write plans or share information. They also get better at their job by remembering specific skills or facts. They might have coworkers that they delegate to and collaborate with.
Agentic harnesses provide the scaffolding for models to do all of that. They provide the ability to use files as scratch pads to store notes, create to-do lists to manage a series of tasks, and operate on your computer using the same kinds of tools that software engineers use. They also let you control which tools a model uses, letting the model operate relatively safely.
More recently, agentic harnesses allow models to make their own “coworkers” to help them out. A model working in an agentic harness is often called an agent, and those coworkers are subagents.
Agents can now manage dozens of subagents working in parallel–one might brainstorm approaches, one might do background research, one might write an article, and two might review the article from different perspectives. All the while each one could use tools, keep notes, perform reasoning, etc.
When models can record notes, manage to-do lists, and work in groups, they are capable of managing larger and larger tasks without forgetting what they were doing or getting off track. Just like people!
The Hugging Face hack was orchestrated by an agent operating a swarm of subagents.
This brave new world
And that’s how we have gotten to where we are today. Models are trained more and more on complex, difficult, and domain-specific tasks that previously only human experts could complete. In tandem, AI companies used RLVR to teach their models to do bigger and bigger tasks more and more effectively with agentic harnesses. AI has accelerated humans like nothing before in our history. Agents can now help humans cure diseases and reach new frontiers of science. They can also hack almost anything and help create bioweapons.
So if you’ve still been using AI as a chatbot, it’s time to step into a brave new world. Go ahead and prompt “write me an app that keeps track of my kids’ schedules,” or “make a video game for me that simulates riding a pirate ship,” or “plan and book a four day trip to Hawaii for me and my kids.” Grab a cup of coffee and prepare for the future while your agent takes over.3
In fact, Google had its own internal chatbot that resembled ChatGPT about a year before ChatGPT launched, but it never launched the chatbot.
Specifically, it used reinforcement learning with human feedback (RLHF) and supervised fine-tuning (SFT). Both are ways of making the desired outcome probable.
You may need to install a browser extension or desktop app and/or use a paid version of common products for these prompts to work.