Building NumPy Arrays from CSV Files, Faster than Pandas

Explore a powerful new approach to converting CSV files into NumPy arrays in this 27-minute PyCon US talk. Dive into the development of delimited_to_arrays(), a C extension designed to outperform Pandas read_csv() while offering full configuration options of Python's csv.reader(), optional type discovery for columns, and support for all NumPy dtypes. Learn about the implementation's architecture, which efficiently collects Unicode code points per column, converts them to C-types, and writes them into NumPy arrays with minimal PyObject creation or reference counting. Discover how this solution, incorporated into the StaticFrame library, achieves significant performance advantages over Pandas across various DataFrame shapes and type heterogeneity. Gain insights into the background, design choices, and performance characteristics of this innovative implementation, which builds upon the 20-year tradition of extending csv.reader() to meet modern data processing needs.

Talks - Christopher Ariza: Building NumPy Arrays from CSV Files, Faster than Pandas