This event has ended. Create your own event → Check it out
This event has ended. Create your own
View analytic
Wednesday, June 1 • 5:20pm - 6:10pm
Building a Machine Learning Orchestration Framework on Apache Mesos - Antony Arokiasamy & Kedar Sadekar, Netflix

Sign up or log in to save this to your schedule and see who's attending!

A scaleable and adaptable machine learning platform is essential for an organization to harness the full potential of their data. This talk outlines how Docker, Spark, Hadoop and several other building blocks can be integrated into a machine learning framework on Mesos. We will discuss how our Mesos framework leverages custom executors, framework/status messages and resource attributes to schedule tasks in a multi-tenant environment. A heterogeneous workload of Spark, Python, R & Scala tasks co-exist and run thousands of computations concurrently on an elastic Mesos cluster of hundreds of nodes. We will demonstrate how a simple user interface and a DSL allows researchers to design, test and deploy resilient machine learning pipelines.

avatar for Antony Arokiasamy

Antony Arokiasamy

Senior Software Engineer, Netflix
Antony is currently an engineer on the Personalization Infrastructure team at Netflix. His primary focus is on building various big data infrastructure components around Spark and Mesos that help algorithmic engineers at Netflix to innovate faster and improve personalization for our members. In the past, he has built machine learning pipelines at Salesforce and AgileOne.

Kedar Sadekar

Sr. Software Engineer, Netflix Inc
I am a Senior software engineer on the Personalization Infrastructure team at Netflix that builds scalable, distributed computing systems for the algorithmic engineers that help improve member personalization. | | Interested in speaking about ML infrastructure challenges and solutions. Also happy to talk about Meson: our general purpose workflow orchestration and scheduling framework that we built to manage ML pipelines that execute... Read More →

Wednesday June 1, 2016 5:20pm - 6:10pm
Chasm Creek

Attendees (43)