Reading in Bloomberg about the increase in memory costs for Nvidia servers, I had a lot of questions. So I spent some of the morning improving my understanding of the bottlenecks to the AI buildout. We’re reading about the AI server-farm backlash, which is one bottleneck to the dreams of those who buy Irish castles and those who hope to, but there are plenty of other pain points (jargon!). Here’s the bottleneck list if you’re interested in this story, with source-story links embedded.
Chips: Can buyers get enough Nvidia processors, and at what price? The processor is the part doing the main mathematical work: turning language, images, video, and code into numbers, then running enormous calculations on those numbers. Nvidia gets our attention because its chips have become the default machinery for training and running the biggest AI models. It also sells whole server systems around those chips. If Nvidia systems get more expensive or harder to get, the cost of building large AI services rises.
Memory: Can those processors get enough specialized high-speed memory, or does memory scarcity push the whole server price up? The processor is fast, but it needs constant access to the model’s data. If the processor has to wait for data, the expensive chip is partly idle. That is why AI systems need specialized memory close to the processor. If that memory gets scarce, the whole machine gets more expensive.
Humanoid robot crashes into reality, breaks in half recently during training for Beijing competition. (Photos: @pianhangx/X) Packaging: Can the processor and memory be assembled close together fast enough to meet demand? “Packaging” sounds like cardboard, but in chips it means the precise physical assembly that connects the processor, memory, wiring, and other components so they can work as one unit. For AI, the processor and memory have to sit extremely close together because data has to move fast. If there are not enough factories that can do this advanced assembly, Nvidia and its customers can have chips and memory in theory but still not enough finished AI systems.
Networking: Can thousands of processors be connected so they behave like one giant computer instead of expensive isolated machines? This means racks of servers, miles of cables, specialized switches, optical links, and software that lets thousands of chips share work without waiting on each other. If the connections are too slow or unreliable, the processors sit around waiting, and the company has paid for a giant computer that cannot fully act like one.
Power: Can the data-center site get enough electricity to run continuously? An AI data center can need electricity at the scale of a town or small city. No juice, no glorious future.
Grid gear: Can utilities get the transformers, switchgear, substations, and cables needed to physically deliver that electricity? AI is a nest of requirements, one of which has its own nest of requirements. Electricity has to pass through heavy equipment that changes voltage, routes power, and protects the system from failures. Some of that equipment now has long wait times. A data center can have a power contract and still be stuck waiting for the machinery that connects it to the grid.
Cooling: Can the building remove the heat from dense racks of AI servers without forcing the chips to slow down or fail? AI chips produce a lot of heat because they are doing nonstop calculations. If the heat is not removed, the chips throttle, crash, or wear out faster.
Water: If the cooling system uses water, can the project justify that demand to local communities already worried about drought, rates, or growth? Some data centers use water to carry heat away. That can turn an AI project into a local water fight, especially in dry regions or fast-growing communities where residents already worry about wells, reservoirs, farms, and utility bills.
Permits: Will local governments, utility regulators, and residents approve the data center, power upgrades, transmission lines, and water use?
Money: If servers, power, land, cooling, debt, and construction all get more expensive, does the buildout still produce enough revenue to justify itself? The AI boom depends on companies believing that future AI revenue will pay for today’s giant infrastructure bills. If the physical parts get more expensive at the same time investors start asking harder questions about profits, some projects get delayed, scaled down, or canceled.
This column is a real public service. Thanks for doing the heavy lifting.
Great questions, John, and they all need to have positive answers before moving forward... not just some of them. Thank you for outlining the scope.