Climbing the Summit with Open Source and POWER9

Explore the journey of creating the world's fastest supercomputer, Summit, powered by POWER9 technology in this 43-minute conference talk from linux.conf.au. Dive into the technical challenges, architectural innovations, and organizational strategies that led to the development of this groundbreaking system. Learn about the major shifts in POWER9 design, including new interrupt controllers, memory management architectures, and advanced IO capabilities like NVLink and OpenCAPI. Discover how open source software played a crucial role in bringing POWER9 from concept to reality, and gain insights into the complex process of building and managing a supercomputer of this scale. Understand the applications of Summit in various scientific fields, including astrophysics, materials science, cancer research, and systems biology. Get a behind-the-scenes look at the timeline, hardware specifications, and software development process that made Summit possible, and explore the future of supercomputing in the post-Moore's Law era.

Intro
maybe even the fastest in the world?
Who wants these machines?
OAK IBM POWER
Intel x86
Summit: Science research Astrophysics Materials Cancer Research Systems Biology
Titan 2012 27
Metric household
Summit: 13 MegaWatts
Summit: USD $200 Million
550 households
1 Sydney house
Summit: 300 km of cables
Sierra: National Nuclear Security Administration's Stockpile Stewardship Mission
How do you build this thing?
IBM 2 computers: • Infrastructure • Compute
POWER8 based?
100Gbps Networking
Mellanox CX-5
Hybrid approach CPUS + GPUS
Compute: Witherspoon AC922
How do we build them?
Timelines?
Sierra release: December 2017
Infrastructure nodes are first
Linux • Firmware • Systems • GPU interfaces
24 Core SMT4
8 Billion transistors
POWER9 is major refresh POWER
Major Architectural changes: • Radix/Linux Based MMU • New interrupt controller • Direct attach DDR4 DIMMs
New Slice Microarchitecture
First through 14nm fab
POWER9 chip development
Minor releases too
DD1: January 2017
Planning for Linux and Firmware
Design: Radix MMU
Radix MMU: • Simpler • Better performance • KVM allocations
Simulation: • Functional • Cycle Accurate
Teach Linux basic feature
Bringup: Everything is broken
Get Linux up
Bringup: • Identify issues • Work around • Get out of the way • Find real fix
Develop items that need real hardware
Testing • More systems • Systems getting more sophisticated • Devs - Machines futher separated
Release: Yay!
Staged release
POWER9 not backwards compatible with POWERS
IBM - RedHat strong relationship
IBM & RedHat partnered on RHEL7 for POWERS
Deliver Linux to customers
End of Moore's Law
Drive accelerators
Binary Linux kernel driver
Helped prove out: • Link training • Firmware
Coherent memory
CUDA Unified memory
Design • IOMMU looks like PCle ATS • IOMMU directly uses Radix MMU
Simulation with P9
Bringup: March 2017
Testing: Data integrity
Baseboard Management Controller
Little computer that turns on your big computer
Firmware?
Infrastructure nodes: Supermicro based BMC
Compute node OpenBMC
Compute nodes first OpenBMC release
Like a distro
Features: • On/Off • Monitor
Solutions
Pervasive
So how did it end up?
Fastest computer in the world?