Building an Instant-On Serverless Platform for Large-Scale Data Processing Using Ray

Explore the development of AWS Glue for Ray, a serverless platform for large-scale data processing, in this 14-minute conference talk. Learn how AWS Glue integrated Ray.io to enable distributed Python workloads and scale data integration tasks. Discover the implementation of Ray's core APIs, distributed collection APIs, and the integration of Modin for efficient ETL operations on massive datasets. Gain insights into the innovations made in cluster management, demand-based autoscaling, and the use of ARM-based platforms with IPv6 addressing. Understand how this serverless Ray platform offers an instant-on, interactive, and user-friendly solution for data engineers working with distributed Pandas at scale.

Building an Instant-On Serverless Platform for Large-Scale Data Processing Using Ray