Numerical Stability of Algorithms at Extreme Scale and Low Precisions

Explore the challenges and solutions in numerical stability of algorithms at extreme scale and low precisions in this 45-minute lecture by Nicholas J. Higham. Delve into the evolving landscape of computer architectures and the approach to exascale computing, examining the implications of using different floating-point precision formats. Investigate how to maximize accuracy in large-scale matrix computations, and learn about innovative techniques to overcome traditional rounding error bounds. Discover the power of blocked algorithms, extended precision registers, and fused multiply-add operations in reducing error constants. Gain insights into probabilistic rounding error analysis and its potential to provide more optimistic error estimates. Examine real-world applications, including deep learning and linear systems, and understand the impact of stochastic rounding on numerical stability. This comprehensive lecture covers topics from TOP500 supercomputers to the latest developments in error analysis, providing a thorough understanding of numerical stability in modern computing environments.

Intro
TOP500: June 2022 Frontier at Oak Ridge AMD EPYC 64C 2GHz, AMD Radeon Instinct GPU. 8,730,112 cores
Growth of Problem Size in TOP500
Today's Floating-Point Arithmetics
Backward Error Analysis for LU Factorizatia
Low Precision in Deep Learning
The (Partial) Explanation
Blocked Inner Products: 2 Pieces
Block Summation
FABsum Error Bound
Random Uniform (0, 1), b = 128, fp32
Extended Precision Registers
Mixed Precision Block FMA
Block FMA Hardware
Error Analysis of Block FMAS Blanchard, H, Lopez, Mary, & Pranesh (2020). Analysis of algs for matrix mult c - AB based on block
NVIDIA V100
Probabilistic Error Analysis Rounding error bounds above are worst-case.
Statistical Effects
Standard Tool for Rounding Error Analysis Theorem If | sufori - 1 n and nu 1 then
Assumptions for Probabilistic Analysis
Probabilistic Analysis Theorem (Connolly, H & Mary, 2021)
Linear Systems
Real-Life Matrices
Probabilistic QR Error Bound Theorem (Connolly & H, 2022) Under Model Mand a technical assumption, for the
Stochastic Rounding Forsythe (1950). .... Croci et al. (2022).
Stagnation Harmonic sum 1/k in fp16.
Random Data
Putting It All Together
Conclusions