Loading…
MesosCon NA 2016 has ended
Back To Schedule
Wednesday, June 1 • 5:20pm - 6:10pm
Building a Machine Learning Orchestration Framework on Apache Mesos - Antony Arokiasamy & Kedar Sadekar, Netflix

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Tweet Share
A scaleable and adaptable machine learning platform is essential for an organization to harness the full potential of their data. This talk outlines how Docker, Spark, Hadoop and several other building blocks can be integrated into a machine learning framework on Mesos. We will discuss how our Mesos framework leverages custom executors, framework/status messages and resource attributes to schedule tasks in a multi-tenant environment. A heterogeneous workload of Spark, Python, R & Scala tasks co-exist and run thousands of computations concurrently on an elastic Mesos cluster of hundreds of nodes. We will demonstrate how a simple user interface and a DSL allows researchers to design, test and deploy resilient machine learning pipelines.
Speakers
avatar for Antony Arokiasamy

Antony Arokiasamy

Senior Software Engineer, Netflix
Antony is currently an engineer on the Personalization Infrastructure team at Netflix. His primary focus is on building various big data infrastructure components around Spark and Mesos that help algorithmic engineers at Netflix to innovate faster and improve personalization for our... Read More →
KS

Kedar Sadekar

Sr. Software Engineer, Netflix Inc
I am a Senior software engineer on the Personalization Infrastructure team at Netflix that builds scalable, distributed computing systems for the algorithmic engineers that help improve member personalization. Interested in speaking about ML infrastructure challenges and solutions... Read More →

Meson MesosCon pdf
Wednesday June 1, 2016 5:20pm - 6:10pm PDT
Chasm Creek
  Frameworks, Intermediate

Attendees (43)