OpenAI and Microsoft are apparently planning to build a $100 billion data center codenamed Stargate. We discuss how this compares to existing data centers and other planned investments into AI centric infrastructure. They don't seem to have enough time to design special purpose networking and other hardware, but it is a massive investment compared to other plans.

We discuss the design problems you have to solve when creating a data center. You need to power it, you need to make sure it stays cool, provide networking, and provide resilience and redundancy for everything. We also discuss how Google data centers are a little different from the norm.

Finally, we discuss the AI chips that could be present in a data center. Primarily, this means Nvidia gpus or Google TPUs. There are implications for the software stack and ultimate usability of the system.

Why Does OpenAI Need a 'Stargate' Supercomputer? Ft. Perplexity CEO Aravind Srinivas
   • Why Does OpenAI Need a 'Stargate' Sup...  

Making AI accessible with Andrej Karpathy and Stephanie Zhan
   • Making AI accessible with Andrej Karp...  

Google AI Infrastructure Supremacy: Systems Matter More Than Microarchitecture
https://www.semianalysis.com/p/google...

Microsoft & OpenAI consider $100bn, 5GW 'Stargate' AI data center - report
https://www.datacenterdynamics.com/en...

ELI5 : How's it that just 400 cables under the ocean provides all the internet to entire world and who actually owns and manages these cables
  / eli5_hows_it_that_just_400_cables_under_th...  

Newmark: US data center power consumption to double by 2030
https://www.datacenterdynamics.com/en...

Data Centres and Data Transmission Networks
https://www.iea.org/energy-system/bui...

Understanding Data Center Costs and How they Compare to the Cloud
https://granulate.io/blog/understandi...

Cost estimate to build and run a data center with 100k AI accelerators - and plenty questions
  / cost_estimate_to_build_and_run_a_data_cent...  

Amazon Bets $150 Billion on Data Centers Required for AI Boom
https://www.bloomberg.com/news/articl...

The world’s top data centre investors
https://www.fdiintelligence.com/conte...

#ai #datacenter #openai

0:00 Intro
0:26 Contents
0:33 Part 1: Data center gold rush
0:46 Server racks and data centers
1:28 What about spending $1 billion?
1:47 50,000 AI accelerators
2:12 $7 trillion previous plan
2:34 OpenAI asks for $10 billion
2:50 OpenAI asks for $100 billion
3:13 Codename Stargate, from science fiction show
3:37 100,000 GPU limit
4:03 Amazon invests $148 billion
4:29 Google has significant data center investment
5:10 Do we actually need this much compute?
5:57 Part 2: So you want to build a datacenter
6:46 Design challenge 1: power consumption
7:11 Proportion of global power consumption
8:09 Collapse of carbon credit market
8:41 Data centers use prepurchased renewable electricity
9:27 Design challenge 2: Cooling
9:44 Power for cooling exceeds power for servers
10:06 Google runs hotter data centers
10:40 Design challenge 3: Networking
11:21 Fiber optic cables
12:26 Intra-rack and inter-gpu networks
13:20 Design challenge 4: Resilience and redundancy
14:36 Don't rely on a single data center
15:22 Stargate has a single region design
15:49 Part 3: The hardware secret sauce
16:25 Hardware stacks
16:28 Nvidia has a monopoly due to CUDA
16:59 Nvidia charges very high prices
17:23 Consumer-grade GPU clusters are cheaper
17:51 Google has TPUs as a GPU alternative
18:29 TPU microarchitecture
19:02 TPU network is a 3D torus
19:59 Software stacks
20:13 PyTorch and Tensorflow
20:52 Computational graph representation
21:25 Python reflection to create graph
21:59 XLA compiler from Google
22:26 Leverages LLVM compiler technology
23:04 Example: LLVM also used in web browsers
23:22 Will Stargate use their own chips, network?
24:19 AI chips use a lot more power than usual
24:45 Implications of building Stargate
25:12 Conclusion
26:00 AI-specific hardware suppliers
26:53 Outro