The AI revolution has been pretty mind blowing, but there’s a dark side to it that you might not realize. All the servers need to train and run these generative AI models, requiring unbelievable amounts of energy. But it’s not just that, it’s the rate at which the sector is growing. It’s a gold rush, with hundreds of companies wanting to stake their claim. And when you have a gold rush, all stewardship, and responsibility go out the window.
And don’t forget that already have massive data centers, we’re already trying to move internal combustion cars over to EVs, and use heat pumps to use more electricity for heating and cooling. Our aging grid is already strained, and this may be one of the hottest summers in history, so what will the rise of AI mean for our power grids? Let’s figure this out together. I’m Ricky and this is two bit da vinci.
Source: TIME
Background
I think Microsoft President Brad Smith summed it up perfectly when he said,
“In 2020, we unveiled what we called our carbon moonshot. That was before the explosion in artificial intelligence,”
But last year, Microsoft took a massive step backwards in that regard by increasing its greenhouse gas emissions by 30%, due largely to its AI pursuits.
“So in many ways the moon is five times as far away as it was in 2020, if you just think of our own forecast for the expansion of AI and its electrical needs.”
So just how much electricity does the AI revolution require? What if i told you that NO ONE KNOWS. Companies that pride themselves on transparency are surprisingly tight lipped about how much energy their AI datacenters consume. Plus its actually really hard to calculate since the energy consumption is widely variable.
But that doesn’t mean we can’t try to make an educated guess. Currently data centers consume between 1- 1.5% of all the energy on Earth, according to the International Energy Agency. So much electricity did all of Earth use in 2023?
[do a bit where i do this on camera] Actually let’s ask chatGPT.
Ok looks like about 28,500 TWh. Now if I do a search on Google, I’m seeing yah, about 28,000 or so TWh. But did you know, that chatGPT request consumes from 10 to 25x as much energy as a google search, depending on who you ask.
And get this, for a 20 query conversation with ChatGPT, you’d consume roughly half a liter of water, which almost doesn’t make sense, but I’ll get back to that in a minute.
According to Stanford University, fine-tuning GPT3, took 1,287 MWh (Megawatt hours) of electricity, and this technology consumes more energy with each new step in its development.
GPT-2, for instance, has 1.5 billion parameters, while GPT-3, has 175 billion parameters. This complexity leads to amazing use cases like image and video generation, but comes at the cost of increasing energy consumption.
Consulting firm Gartner calculated that at this rate, AI will account for up to 3.5% of global electricity demand by 2030. That’s as much as agriculture and forestry combined, or twice as much as the entire country of France.
That means based on the 20 year average of 2% increase in yearly energy use worldwide, the entire world would use 33,330 TWh of electricity, and AI alone would consume 1,167 TWh.
If we looked at electricity use by country, for 2022, the latest data I could find, that would put AI electricity use behind only the countries of China, the United States and India. It would rank 4th! (data)
Put another way, according to the EIA the average home in the united states consumes 883kWh per month, or roughly 10.6MWh per year. (link) that means the electricity used in AI would be enough to power 110M of the 131M households in all of the United States! (link)
The troubling part is that like I mentioned, this is a gold rush, which means companies are going to be battling for their piece of the pie, consequences be damned.
The age of corporate responsibility, where companies were growing their data centers and energy footprint along with their sustainability plans, with new energy generation, sadly might be over. New wind and solar farms won’t be able to keep up with this level of massive growth.
All this data doesn’t just consume energy, it needs to be stored on hard drives, kind of forever, and this can include our personal information. It’s why i’ve been a member of our sponsor this week Delete Me for over a year.
;;;
And when i say Gold Rush I mean it. For example, look at the king of internet search. Google made $175B just from search. If the internet is the greatest marketplace of all time, Google is the gate keeper.
Their dominance in search has been viewed as an insurmountable moat for so long, that it’s become the target of anti trust legislation.
But for the first time in 2 decades their 175B pie, might be up for grabs. Remember earlier when I tried googling something, and then asked chat GPT? Well using AI to find answers might very well use the same indexing and search optimizations that have made Google a Trillion dollar company. But it bypasses all its ads. Just look at this, this is what I get when I search for “solar panels”
The entire first page, is sponsored ads, paid for by companies, and their “popular products” section.
But with chat GPT this entire money printing enterprise is entirely bypassed. Google knows this, and that’s why they are investing BILLIONS in new AI data centers, like their $1 billion data center campus in Kansas City, Mo, or their $576 million data center project in Cedar Rapids, Iowa.
The rise of AI is going to place a major strain on an already strained grid. And we have to remember that other electrifications initiatives are already underway, like the rise in EV sales. EVs accounted for 5% of all cars sold in the U.S. in 2022 (link), and nearly 14M EVs were sold worldwide in 2023 (link). The luxury of EV growth has been its slow steady rise, giving companies and municipalities time to invest in the grid, increase energy generation and install charging stations. But AI is going to be different, because AI is a gold rush.
Imagine if your boss said this month, you can work as many hours as you want, and I’ll pay you 10x your usual hourly rate. Wouldn’t it be tempting to stop going to the gym, skip family time, cancel the meditation classes, and just work?
Now big Tech has been a little quiet when it comes to the energy demands of all their new AI datacenter ambitions. Companies like Google, Meta, Amazon and Microsoft, all have environmental impact goals in their charter. They say they buy enough wind, solar or geothermal power every time a big data center comes online to offset its resulting emissions.
But there’s a problem with these renewable energy contracts. Big AI is pulling energy from the same grid as everyone else, and by buying green energy contracts, they’re using up much of the finite amount of green energy generation available. To then fulfill the needs of other customers, utilities are then filling out their energy portfolio with fossil fuel expansion, or in some cases, by extending the life of coal power plants, the dirtiest form of energy, instead of shutting them down as planned.
So while Big tech companies may claim in their marketing that their new AI data centers are clean, that comes at the expense of everyone else getting their energy from dirtier sources.
One other trend I’ve noticed more and more, is moving from more general to more custom large language models. Many companies are taking these LLMs and training it only on relevant datasets based on their industry. If you ever see an ad for a new AI platform tailored for enterprise, or for image generation, or for education, that’s a new model that requires its own training. It’s like having a general contractor who can do everything, vs. hiring an electrician for the wiring, a plumber for the water and sewer, a framer for the foundation, and a stone mason for the brickwork. Each new custom model will require all that training over and over again.
But there is also some good news.
Have you ever heard the expression that the ones who truly get rich during a gold rush are the ones who sell shovels? Well in this AI analogy, the shovel salesmen are Nvidia, a company that has absolutely skyrocketed in recent months, due to their absolute dominance in AI tailored GPUs for data centers.
NVIDIA reportedly has 98% market share of the AI GPU market, and shipped almost 4 million AI GPUs in 2023. And this number is only going to be climbing. (link)
NVidia is hard at work on constantly improving their hardware, and improving the performance per watt of their GPUs. For example their latest B200 GPU based on their blackwell architecture offers up to 20 petaflops of FP4 horsepower from its 208 billion transistors. And NVidia says Training a 1.8 trillion parameter model would have previously taken 8,000 Hopper GPUs and 15 megawatts of power. Today, Nvidia’s CEO says 2,000 Blackwell GPUs can do it while consuming just four megawatts.
But these GPUs don’t operate in a vacuum, and are clustered together in massive server racks. NVidia says that in a previous server cluster of just 16 GPUs, nearly 60% of their time was spent communicating with one another, leaving only about 40% to actually perform compute workloads. Now with a whole system approach, Nvidia has built an entirely new network switch chip, with 50 billion transistors and some of its own onboard compute. These optimizations will greatly increase the amount of actual AI workload performed, per watt of energy consumed.
And in this Gold Rush, Nvidia isn’t alone, with competitors like Intel trying to catch up with their Gaudi 3 line of data center GPUs. Or the Instinct™ MI300 line from AMD.
And this really matters, because the limiting factor for the growth of AI Data Centers might simply come down to supply chains, and how quickly companies like Nvidia can produce their AI GPUs.
Another really fascinating area of study is analog computers, which sounds counterintuitive. But once a model is trained, with some clever electrical engineering, analog processors could run AI models at a fraction of the energy costs. If you’re curious about this, we have a full deep dive video on analog computers that is super fascinating which we’ll link here, and in the video description.
And check this out. A fan of the channel sent me this article about research being done by the University of California Santa Cruz, UC Davis, LuxiTech, and Soochow University, that could have deep implications for the environmental impact and operational costs of AI.
Huge thanks to our viewer Tony Herstell for sharing this. At its core, what makes GPUs so good at rendering video games, performing transforms, 3D, and light tracing, is its ability to perform complex mathematical operations quickly and in parallel. This same ability makes it great at Matrix multiplication which is at the core of neural network operations. Definitely check out our video on analog computers for a deeper dive on how they work.
But these researchers claim to have developed a new way to run AI language models more efficiently by eliminating matrix multiplication from the process. This fundamentally redesigns neural network operations that are currently accelerated by GPU chips.
This would be like a breakthrough energy source that needed gold in order to produce, and then discovering a new way to do the same thing, using something much more abundant like aluminum. That would profoundly change the equation for what future AI energy consumption might look like. Sound off in the comments if you want me to cover this cool research in a full video, because it’s beyond the scope of this video.
Also a company called Soluna we’ve covered in the past, which we’ll link here and the description, sees this challenge coming and has their own novel solution. Surprisingly, many wind and solar farms have to stop exporting energy at times when there isn’t enough demand, its one of the great challenges of renewable and intermittent energy. So Soluna has a pilot plant we toured in Amarillo Texas, where they have co-located a large computer cluster that fires up only during these times. This means we could run AI training models during times of excess energy, better utilizing that energy instead of wasting, and remove the need on the demand side. Plus it levels out the revenue model for these renewable energy operators making it more lucrative and incentives further investment.
And remember when I mentioned the water use? Well that comes from cooling, where evaporative coolers are often used to draw out the massive amounts of heat these data centers provide, instead of running heat pumps which consume significantly more energy. But whether we use heat pumps for cooling, or evaporate water, it all comes at a cost. Leave me a comment if you want to see a video on how this “waste heat” which is one of the most fundamental forms of energy, could actually be valuable, and how we could even produce energy from it.
We live in an amazing time, with so many people and companies working on so many things. AI is here to stay, but should we be optimistic, well what it might mean for society would be a topic for another day, but in terms of its energy use and its impact on our grids, I’d say, we’ve always found a way, and I think this is no different.
Like I mentioned we have a few videos that are highly related, and would make for excellent viewing at this one.
