Building an Instant-On Serverless Platform for Large-Scale Data Processing Using Ray

Explore the development of AWS Glue for Ray, a serverless platform for large-scale data processing, in this 14-minute conference talk. Learn how AWS Glue integrated Ray.io to enable distributed Python workloads and scale data integration tasks. Discover the implementation of Ray's core APIs, distributed collection APIs, and the integration of Modin for efficient ETL operations on massive datasets. Gain insights into the innovations made in cluster management, demand-based autoscaling, and the creation of an instant-on, interactive serverless Ray platform. Understand how this solution addresses customer needs for scaling Python workloads over large datasets, utilizing ARM-based platforms and IPv6 addressing for workers.

Building an Instant-On Serverless Platform for Large-Scale Data Processing Using Ray