- The newest leap in AI did not come from bigger training. It came from letting models think longer when you ask, spending extra compute to reason before answering.
- That thinking happens in hidden tokens you never see but pay for, so a single hard question can cost many times more than a normal one.
- This flipped AI's cost structure: in 2026, running models (inference) now burns more compute and money than training them, which is reshaping chips and margins.
The Setup
For years, the way to make AI smarter was to make it bigger: more data, more parameters, larger training runs. That curve flattened. The new gains came from a different idea, let the model think longer when you ask it something, instead of answering on reflex. The industry calls it test-time compute, and it quietly rewired both the technology and the economics.What Test-Time Compute Actually Is
A normal model answers in one pass, like blurting the first thing that comes to mind. A reasoning model pauses and works through the problem, trying approaches, checking itself, backtracking, before it replies. It does this by generating a long internal chain of thought, hidden reasoning tokens that never show up in the chat bubble. More thinking means more tokens and better answers on hard problems. It is the difference between System 1, fast and instinctive, and System 2, slow and deliberate.How They Learn to Think
The trick is reinforcement learning. You take a base model, drop it into an environment that rewards correct answers to hard problems, and let it learn that reasoning step by step pays off. Models like OpenAI's o-series and DeepSeek's R1 were built this way. The striking part: DeepSeek-R1 matched a far more expensive model partly by generating 10 to 100 times more tokens per question. It traded raw size for raw thinking time.Why It Costs So Much
Here is the catch most people never see. Those hidden thinking tokens are real compute, and you are billed for them. A reasoning model can consume on the order of 150 times more compute than a plain answer, and on some hard tasks the energy cost runs over 100 times higher. So the smart answer is slower and dramatically more expensive, and the price scales with how hard the question is. Your invoice now depends on how much the model decided to think.Why This Changes the Whole Business
This is the big one. Training a model is a large one-time cost. Running it is a cost you pay forever, every single query. Once a model is called billions of times a day, the running cost dwarfs the training cost. In early 2026, for the first time, the world spent more on AI inference than on training, with inference now around two-thirds of all AI compute. That is why Nvidia and others are pivoting hard toward inference economics, cost per token rather than just training horsepower.What It Means For Investors
Follow the compute. The AI trade is shifting from who trains the biggest model to who serves answers cheapest at scale. That favors inference-optimized chips, efficient serving, and anyone who can cut cost per token. It also makes AI gross margins a real question, because every clever reasoning answer eats variable cost. And watch the overthinking problem: more thinking is not always better, it hits diminishing returns and can even hurt, so efficiency becomes the edge, not raw thinking.FAQ
Is a reasoning model always better than a normal one?No. It helps a lot on hard, multi-step problems like math and code, but on simple questions it just burns time and money for no gain. On some knowledge tasks it does not help yet. Match the tool to the task.
Why can't I see the model thinking?
Most labs hide the chain of thought, partly to protect their method and partly because it is messy. You still pay for those hidden tokens, which is why a short answer can carry a surprisingly large bill.
What is the investor takeaway in one line?
Inference is the new battleground. The money is moving from training the model to serving it, so the winners will be whoever delivers a good answer at the lowest cost per token.