Scaling Python for Machine Learning: Beyond Data Parallelism • Holden Karau • GOTO 2023

From the creator

This presentation was recorded at GOTO Chicago 2023. #GOTOcon #GOTOchgo https://gotochgo.com Holden Karau - Open Source Engineer at Netflix @HoldenKarau RESOURCES http://www.holdenkarau.com https://twitter.com/holdenkarau https://www.twitch.tv/holdenkarau https://tech.lgbt/@holden https://www.distributedcomputing4kids.com ABSTRACT Data Parallelism can be amazing and it frees us from so many fiddly complicated tasks (like dealing with locks). On the other hand, as training large machine learning models becomes increasingly popular, we're seeing the need to move beyond purely data-parallel techniques. Depending on recompute exclusively for failure is no longer sufficient as our operations are not idempotent. In this talk we will look at Spark, Dask, and Ray in the context of scaling machine learning models and how you can take advantage of other types of distributed parallelism (including the actor model for managing model weights during training). [...] TIMECODES 00:00 Intro 04:16 Probable (relevant) biases 06:14 Quick refresher on data parallelism 11:52 What do (distributed) tasks look like? 12:49 Dask distributed tasks 13:42 Ray distributed tasks 14:03 How are they different & same? 18:41 Task fault tolerance 21:53 Does Spark have tasks & actors? 27:39 Ray Diagram 31:33 Ray actor fault tolerance 33:43 What's up with Ray & Netflix? 34:39 Dask actor fault tolerance 37:28 Outro Download slides and read the full abstract here: https://gotochgo.com/2023/sessions/2596 RECO

Published 2023-08-28 · Watch on YouTube ↗
Availability and playback restrictions are controlled by the original channel.